Skip to content

Intercept messages to consumers and add intercept exception #8129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Sep 29, 2020

Conversation

zymap
Copy link
Member

@zymap zymap commented Sep 25, 2020

(If this PR fixes a github issue, please add Fixes #<xyz>.)

Fixes #

(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)

Master Issue: #

Motivation

Explain here the context, and why you're making that change. What is the problem you're trying to solve.

Modifications

Describe the modifications you've done.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Sorry, something went wrong.

codelipenghui and others added 6 commits September 21, 2020 21:12

Verified

This commit was signed with the committer’s verified signature.
zymap Yong Zhang
)

Verified

This commit was signed with the committer’s verified signature.
zymap Yong Zhang

Verified

This commit was signed with the committer’s verified signature.
zymap Yong Zhang

Verified

This commit was signed with the committer’s verified signature.
zymap Yong Zhang

Verified

This commit was signed with the committer’s verified signature.
zymap Yong Zhang
* Don't expose filter chain to interceptors.

* Add RequestWrapper

Verified

This commit was signed with the committer’s verified signature.
zymap Yong Zhang
@zymap
Copy link
Member Author

zymap commented Sep 25, 2020

/pulsarbot run-failure-checks

1 similar comment
@zymap
Copy link
Member Author

zymap commented Sep 27, 2020

/pulsarbot run-failure-checks

Verified

This commit was signed with the committer’s verified signature.
zymap Yong Zhang
@zymap
Copy link
Member Author

zymap commented Sep 28, 2020

/pulsarbot run-failure-checks

1 similar comment
@zymap
Copy link
Member Author

zymap commented Sep 28, 2020

/pulsarbot run-failure-checks

@codelipenghui codelipenghui merged commit 73e5bc4 into apache:master Sep 29, 2020
lbenc135 pushed a commit to lbenc135/pulsar that referenced this pull request Oct 3, 2020
)

* Intercept messages to consumers and add intercept exception
* Capture response for broker interceptor.
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Nov 13, 2020
)

* Intercept messages to consumers and add intercept exception
* Capture response for broker interceptor.
codelipenghui pushed a commit that referenced this pull request Nov 3, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Cherry pick from #7473.

#7473 has fix the `Cleanup already deleted namespace topics` issue, but with #8129 involved, changes have been 
changed back.

### Motivation
We are having frequent issues when user removes cluster from the global namespace where broker from removed-cluster fails to unload topic and namespace bundle still loaded with the broker. It happens when broker from removed-cluster receives below error
```
17:38:52.199 [pulsar-io-22-28] ERROR org.apache.pulsar.broker.service.persistent.PersistentReplicator - [persistent://prop/global/ns/tp1][east -> west] Failed to close dispatch rate limiter: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
:
17:38:52.199 [pulsar-io-22-28] WARN  org.apache.pulsar.broker.service.AbstractReplicator - [persistent://prop/global/ns/tp1][east -> west]] Exception: 'org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection' occured while trying to close the producer. retrying again in 0.1 s
:
17:38:52.351 [pulsar-io-22-37] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://prop/global/ns/tp1] Error closing topic
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
```

### Modification
Source Broker should return explicit error-code when producer is already closed and dest-broker from removed-cluster should handle this error and clean up the replicator and topic gracefully.
hangc0276 pushed a commit that referenced this pull request Nov 4, 2021
Cherry pick from #7473.

#7473 has fix the `Cleanup already deleted namespace topics` issue, but with #8129 involved, changes have been
changed back.

### Motivation
We are having frequent issues when user removes cluster from the global namespace where broker from removed-cluster fails to unload topic and namespace bundle still loaded with the broker. It happens when broker from removed-cluster receives below error
```
17:38:52.199 [pulsar-io-22-28] ERROR org.apache.pulsar.broker.service.persistent.PersistentReplicator - [persistent://prop/global/ns/tp1][east -> west] Failed to close dispatch rate limiter: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
:
17:38:52.199 [pulsar-io-22-28] WARN  org.apache.pulsar.broker.service.AbstractReplicator - [persistent://prop/global/ns/tp1][east -> west]] Exception: 'org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection' occured while trying to close the producer. retrying again in 0.1 s
:
17:38:52.351 [pulsar-io-22-37] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://prop/global/ns/tp1] Error closing topic
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
```

### Modification
Source Broker should return explicit error-code when producer is already closed and dest-broker from removed-cluster should handle this error and clean up the replicator and topic gracefully.

(cherry picked from commit 6b3fb41)
eolivelli pushed a commit to eolivelli/pulsar that referenced this pull request Nov 29, 2021
Cherry pick from apache#7473.

apache#7473 has fix the `Cleanup already deleted namespace topics` issue, but with apache#8129 involved, changes have been 
changed back.

### Motivation
We are having frequent issues when user removes cluster from the global namespace where broker from removed-cluster fails to unload topic and namespace bundle still loaded with the broker. It happens when broker from removed-cluster receives below error
```
17:38:52.199 [pulsar-io-22-28] ERROR org.apache.pulsar.broker.service.persistent.PersistentReplicator - [persistent://prop/global/ns/tp1][east -> west] Failed to close dispatch rate limiter: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
:
17:38:52.199 [pulsar-io-22-28] WARN  org.apache.pulsar.broker.service.AbstractReplicator - [persistent://prop/global/ns/tp1][east -> west]] Exception: 'org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection' occured while trying to close the producer. retrying again in 0.1 s
:
17:38:52.351 [pulsar-io-22-37] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://prop/global/ns/tp1] Error closing topic
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
```

### Modification
Source Broker should return explicit error-code when producer is already closed and dest-broker from removed-cluster should handle this error and clean up the replicator and topic gracefully.
lhotari pushed a commit to datastax/pulsar that referenced this pull request Dec 7, 2021
Cherry pick from apache#7473.

apache#7473 has fix the `Cleanup already deleted namespace topics` issue, but with apache#8129 involved, changes have been
changed back.

### Motivation
We are having frequent issues when user removes cluster from the global namespace where broker from removed-cluster fails to unload topic and namespace bundle still loaded with the broker. It happens when broker from removed-cluster receives below error
```
17:38:52.199 [pulsar-io-22-28] ERROR org.apache.pulsar.broker.service.persistent.PersistentReplicator - [persistent://prop/global/ns/tp1][east -> west] Failed to close dispatch rate limiter: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
:
17:38:52.199 [pulsar-io-22-28] WARN  org.apache.pulsar.broker.service.AbstractReplicator - [persistent://prop/global/ns/tp1][east -> west]] Exception: 'org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection' occured while trying to close the producer. retrying again in 0.1 s
:
17:38:52.351 [pulsar-io-22-37] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://prop/global/ns/tp1] Error closing topic
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
```

### Modification
Source Broker should return explicit error-code when producer is already closed and dest-broker from removed-cluster should handle this error and clean up the replicator and topic gracefully.

(cherry picked from commit 6b3fb41)
(cherry picked from commit 434fbcd)
codelipenghui pushed a commit that referenced this pull request Apr 19, 2022
Cherry pick from #7473.

#7473 has fix the `Cleanup already deleted namespace topics` issue, but with #8129 involved, changes have been
changed back.

### Motivation
We are having frequent issues when user removes cluster from the global namespace where broker from removed-cluster fails to unload topic and namespace bundle still loaded with the broker. It happens when broker from removed-cluster receives below error
```
17:38:52.199 [pulsar-io-22-28] ERROR org.apache.pulsar.broker.service.persistent.PersistentReplicator - [persistent://prop/global/ns/tp1][east -> west] Failed to close dispatch rate limiter: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
:
17:38:52.199 [pulsar-io-22-28] WARN  org.apache.pulsar.broker.service.AbstractReplicator - [persistent://prop/global/ns/tp1][east -> west]] Exception: 'org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection' occured while trying to close the producer. retrying again in 0.1 s
:
17:38:52.351 [pulsar-io-22-37] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://prop/global/ns/tp1] Error closing topic
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: Producer was not registered on the connection
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
```

### Modification
Source Broker should return explicit error-code when producer is already closed and dest-broker from removed-cluster should handle this error and clean up the replicator and topic gracefully.

(cherry picked from commit 6b3fb41)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants