Skip to content

[broker] Fix error code returned to client when service unit is not ready #8147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 1, 2020

Conversation

massakam
Copy link
Contributor

Motivation

The other day, when a namespace bundle was unloaded, the broker got the following exception:

18:30:35.309 [ForkJoinPool.commonPool-worker-60] WARN  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:45182][persistent://xxx/xxx/xxx][xxx] Failed to create consumer: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:998)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$13(ServerCnx.java:682)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943)
        at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163)
Caused by: java.lang.RuntimeException: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1016)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:509)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$10(ServerCnx.java:699)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
        ... 12 common frames omitted
Caused by: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        ... 16 common frames omitted

If a ServiceUnitNotReadyException occurs on the broker side, the client should receive a ServiceNotReady error. However, the client received an UnknownError at this time.

The broker was unable to return the correct error code to the client because the ServiceUnitNotReadyException was wrapped by a RuntimeException. Broker returns an UnknownError if an unknown exception is thrown:

private static PulsarApi.ServerError getClientErrorCode(Throwable t, boolean checkCauseIfUnknown) {
if (t instanceof ServerMetadataException) {
return PulsarApi.ServerError.MetadataError;
} else if (t instanceof NamingException) {
return PulsarApi.ServerError.ProducerBusy;
} else if (t instanceof PersistenceException) {
return PulsarApi.ServerError.PersistenceError;
} else if (t instanceof ConsumerBusyException) {
return PulsarApi.ServerError.ConsumerBusy;
} else if (t instanceof UnsupportedVersionException) {
return PulsarApi.ServerError.UnsupportedVersionError;
} else if (t instanceof TooManyRequestsException) {
return PulsarApi.ServerError.TooManyRequests;
} else if (t instanceof TopicTerminatedException) {
return PulsarApi.ServerError.TopicTerminatedError;
} else if (t instanceof ServiceUnitNotReadyException || t instanceof TopicFencedException
|| t instanceof SubscriptionFencedException) {
return PulsarApi.ServerError.ServiceNotReady;
} else if (t instanceof TopicNotFoundException) {
return PulsarApi.ServerError.TopicNotFound;
} else if (t instanceof IncompatibleSchemaException
|| t instanceof InvalidSchemaDataException) {
// for backward compatible with old clients, invalid schema data
// is treated as "incompatible schema".
return PulsarApi.ServerError.IncompatibleSchema;
} else if (t instanceof ConsumerAssignException) {
return ServerError.ConsumerAssignError;
} else if (t instanceof CoordinatorException.CoordinatorNotFoundException) {
return ServerError.TransactionCoordinatorNotFound;
} else if (t instanceof CoordinatorException.InvalidTxnStatusException) {
return ServerError.InvalidTxnStatus;
} else if (t instanceof NotAllowedException) {
return ServerError.NotAllowedError;
} else {
if (checkCauseIfUnknown) {
return getClientErrorCode(t.getCause(), false);
} else {
return PulsarApi.ServerError.UnknownError;
}
}
}

Modifications

Fixed the BrokerService#checkTopicNsOwnership(String) method to throw a BrokerServiceException instead of a RuntimeException.

@massakam massakam added type/bug The PR fixed a bug or issue reported a bug area/broker release/2.6.2 labels Sep 28, 2020
@massakam massakam added this to the 2.7.0 milestone Sep 28, 2020
@massakam massakam self-assigned this Sep 28, 2020
@sijie sijie merged commit abe9222 into apache:master Oct 1, 2020
@massakam massakam deleted the su-not-ready branch October 1, 2020 06:46
lbenc135 pushed a commit to lbenc135/pulsar that referenced this pull request Oct 3, 2020
…che#8147)

### Motivation

The other day, when a namespace bundle was unloaded, the broker got the following exception:
```
18:30:35.309 [ForkJoinPool.commonPool-worker-60] WARN  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:45182][persistent://xxx/xxx/xxx][xxx] Failed to create consumer: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:998)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$13(ServerCnx.java:682)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943)
        at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163)
Caused by: java.lang.RuntimeException: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1016)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:509)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$10(ServerCnx.java:699)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
        ... 12 common frames omitted
Caused by: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        ... 16 common frames omitted
```

If a `ServiceUnitNotReadyException` occurs on the broker side, the client should receive a `ServiceNotReady` error. However, the client received an `UnknownError` at this time.

The broker was unable to return the correct error code to the client because the `ServiceUnitNotReadyException` was wrapped by a `RuntimeException`. Broker returns an `UnknownError` if an unknown exception is thrown:
https://github.com/apache/pulsar/blob/f8b2a2334fb7d2dc5266242a6393c9cc434fba60/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerServiceException.java#L184-L224

### Modifications

Fixed the `BrokerService#checkTopicNsOwnership(String)` method to throw a `BrokerServiceException` instead of a `RuntimeException`.
wolfstudy pushed a commit that referenced this pull request Oct 30, 2020
### Motivation

The other day, when a namespace bundle was unloaded, the broker got the following exception:
```
18:30:35.309 [ForkJoinPool.commonPool-worker-60] WARN  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:45182][persistent://xxx/xxx/xxx][xxx] Failed to create consumer: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:998)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$13(ServerCnx.java:682)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943)
        at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163)
Caused by: java.lang.RuntimeException: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1016)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:509)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$10(ServerCnx.java:699)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
        ... 12 common frames omitted
Caused by: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        ... 16 common frames omitted
```

If a `ServiceUnitNotReadyException` occurs on the broker side, the client should receive a `ServiceNotReady` error. However, the client received an `UnknownError` at this time.

The broker was unable to return the correct error code to the client because the `ServiceUnitNotReadyException` was wrapped by a `RuntimeException`. Broker returns an `UnknownError` if an unknown exception is thrown:
https://github.com/apache/pulsar/blob/f8b2a2334fb7d2dc5266242a6393c9cc434fba60/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerServiceException.java#L184-L224

### Modifications

Fixed the `BrokerService#checkTopicNsOwnership(String)` method to throw a `BrokerServiceException` instead of a `RuntimeException`.

(cherry picked from commit abe9222)
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Nov 13, 2020
…che#8147)

### Motivation

The other day, when a namespace bundle was unloaded, the broker got the following exception:
```
18:30:35.309 [ForkJoinPool.commonPool-worker-60] WARN  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:45182][persistent://xxx/xxx/xxx][xxx] Failed to create consumer: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:998)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$13(ServerCnx.java:682)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943)
        at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163)
Caused by: java.lang.RuntimeException: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1016)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:509)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$10(ServerCnx.java:699)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
        ... 12 common frames omitted
Caused by: org.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNotReadyException: Namespace bundle for topic (persistent://xxx/xxx/xxx) not served by this instance. Please redo the lookup. Request is denied: namespace=xxx/xxx
        ... 16 common frames omitted
```

If a `ServiceUnitNotReadyException` occurs on the broker side, the client should receive a `ServiceNotReady` error. However, the client received an `UnknownError` at this time.

The broker was unable to return the correct error code to the client because the `ServiceUnitNotReadyException` was wrapped by a `RuntimeException`. Broker returns an `UnknownError` if an unknown exception is thrown:
https://github.com/apache/pulsar/blob/f8b2a2334fb7d2dc5266242a6393c9cc434fba60/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerServiceException.java#L184-L224

### Modifications

Fixed the `BrokerService#checkTopicNsOwnership(String)` method to throw a `BrokerServiceException` instead of a `RuntimeException`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/broker release/2.6.2 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants