Fix producer stucks on creating ledger timeout #7319
Merged
+13
−10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
*Motivation*
The `ledgerCreated` flag is passed as ctx to the createLedger callback.
The callback already had the logic on handling `ledgerCreated` flag. But we set the flag to `false`
when timeout happens. It will trigger the following race condition:
a) The createComplete callback is triggered when timeout. But the pending add requests are not error'd out.
b) If the createComplete callback eventually completes, it will see `ledgerCreated` flag is set to true,
so it will cause `checkAndCompleteLedgerOpTask` returns false and exist too early without processing the
pending add requests.
This race condition only happens when creating ledger times out.
```
public synchronized void createComplete(int rc, final LedgerHandle lh, Object ctx) {
if (log.isDebugEnabled()) {
log.debug("[{}] createComplete rc={} ledger={}", name, rc, lh != null ? lh.getId() : -1);
}
if (checkAndCompleteLedgerOpTask(rc, lh, ctx)) {
return;
}
```
*Modification*
The timeout logic shouldn't modify the `ledgerCreated` context. It should let the callback itself to process
the `ledgerCreated` context.
codelipenghui
approved these changes
Jun 20, 2020
merlimat
reviewed
Jun 20, 2020
| @@ -3177,8 +3177,7 @@ protected void asyncCreateLedger(BookKeeper bookKeeper, ManagedLedgerConfig conf | |||
| digestType, config.getPassword(), cb, ledgerCreated, finalMetadata); | |||
| scheduledExecutor.schedule(() -> { | |||
| if (!ledgerCreated.get()) { | |||
| ledgerCreated.set(true); | |||
| cb.createComplete(BKException.Code.TimeoutException, null, null); | |||
| cb.createComplete(BKException.Code.TimeoutException, null, ledgerCreated); | |||
There was a problem hiding this comment.
Wouldn't there be a race condition here (also in existing code) if the creation completes at the same time that the timeout is triggered?
There was a problem hiding this comment.
The callback is synchronized at ManagedLedgerImpl. So there is no race condition here.
There was a problem hiding this comment.
It's not always the case. eg: In createNewMetadataLedger() when we create a new cursor ledger, we pass a lambda and there's no mutex. The logic in checkAndCompleteLedgerOpTask() seems race prone.
I think we should probably use compareAndSet() to manipulate that ledgerCreated flag, which also might be better represented by an enum with multiple states.
Finally (and this is unrelated to the bug here), we should try to cancel the timeout task, once the create operation succeeds or fails before it times out. Otherwise we can get into a state with many timer tasks pending.
There was a problem hiding this comment.
I see. I missed the createNewMetadataLedger operation. I changed it to use CAS.
For canceling the timeout tasks, I will have a follow-up issue to do that.
jiazhai
approved these changes
Jun 20, 2020
|
/pulsarbot run-failure-checks |
merlimat
approved these changes
Jun 22, 2020
rdhabalia
approved these changes
Jun 22, 2020
jiazhai
pushed a commit
that referenced
this pull request
Jun 23, 2020
* Fix producer stucks on creating ledger timeout
*Motivation*
The `ledgerCreated` flag is passed as ctx to the createLedger callback.
The callback already had the logic on handling `ledgerCreated` flag. But we set the flag to `false`
when timeout happens. It will trigger the following race condition:
a) The createComplete callback is triggered when timeout. But the pending add requests are not error'd out.
b) If the createComplete callback eventually completes, it will see `ledgerCreated` flag is set to true,
so it will cause `checkAndCompleteLedgerOpTask` returns false and exist too early without processing the
pending add requests.
This race condition only happens when creating ledger times out.
```
public synchronized void createComplete(int rc, final LedgerHandle lh, Object ctx) {
if (log.isDebugEnabled()) {
log.debug("[{}] createComplete rc={} ledger={}", name, rc, lh != null ? lh.getId() : -1);
}
if (checkAndCompleteLedgerOpTask(rc, lh, ctx)) {
return;
}
```
*Modification*
The timeout logic shouldn't modify the `ledgerCreated` context. It should let the callback itself to process
the `ledgerCreated` context.
* Change to use CAS
(cherry picked from commit a34f693)
jiazhai
pushed a commit
that referenced
this pull request
Jun 23, 2020
* Fix producer stucks on creating ledger timeout
*Motivation*
The `ledgerCreated` flag is passed as ctx to the createLedger callback.
The callback already had the logic on handling `ledgerCreated` flag. But we set the flag to `false`
when timeout happens. It will trigger the following race condition:
a) The createComplete callback is triggered when timeout. But the pending add requests are not error'd out.
b) If the createComplete callback eventually completes, it will see `ledgerCreated` flag is set to true,
so it will cause `checkAndCompleteLedgerOpTask` returns false and exist too early without processing the
pending add requests.
This race condition only happens when creating ledger times out.
```
public synchronized void createComplete(int rc, final LedgerHandle lh, Object ctx) {
if (log.isDebugEnabled()) {
log.debug("[{}] createComplete rc={} ledger={}", name, rc, lh != null ? lh.getId() : -1);
}
if (checkAndCompleteLedgerOpTask(rc, lh, ctx)) {
return;
}
```
*Modification*
The timeout logic shouldn't modify the `ledgerCreated` context. It should let the callback itself to process
the `ledgerCreated` context.
* Change to use CAS
(cherry picked from commit a34f693)
codelipenghui
pushed a commit
to streamnative/pulsar-archived
that referenced
this pull request
Jul 14, 2020
* Fix producer stucks on creating ledger timeout
*Motivation*
The `ledgerCreated` flag is passed as ctx to the createLedger callback.
The callback already had the logic on handling `ledgerCreated` flag. But we set the flag to `false`
when timeout happens. It will trigger the following race condition:
a) The createComplete callback is triggered when timeout. But the pending add requests are not error'd out.
b) If the createComplete callback eventually completes, it will see `ledgerCreated` flag is set to true,
so it will cause `checkAndCompleteLedgerOpTask` returns false and exist too early without processing the
pending add requests.
This race condition only happens when creating ledger times out.
```
public synchronized void createComplete(int rc, final LedgerHandle lh, Object ctx) {
if (log.isDebugEnabled()) {
log.debug("[{}] createComplete rc={} ledger={}", name, rc, lh != null ? lh.getId() : -1);
}
if (checkAndCompleteLedgerOpTask(rc, lh, ctx)) {
return;
}
```
*Modification*
The timeout logic shouldn't modify the `ledgerCreated` context. It should let the callback itself to process
the `ledgerCreated` context.
* Change to use CAS
(cherry picked from commit a34f693)
cdbartholomew
pushed a commit
to kafkaesque-io/pulsar
that referenced
this pull request
Jul 24, 2020
* Fix producer stucks on creating ledger timeout
*Motivation*
The `ledgerCreated` flag is passed as ctx to the createLedger callback.
The callback already had the logic on handling `ledgerCreated` flag. But we set the flag to `false`
when timeout happens. It will trigger the following race condition:
a) The createComplete callback is triggered when timeout. But the pending add requests are not error'd out.
b) If the createComplete callback eventually completes, it will see `ledgerCreated` flag is set to true,
so it will cause `checkAndCompleteLedgerOpTask` returns false and exist too early without processing the
pending add requests.
This race condition only happens when creating ledger times out.
```
public synchronized void createComplete(int rc, final LedgerHandle lh, Object ctx) {
if (log.isDebugEnabled()) {
log.debug("[{}] createComplete rc={} ledger={}", name, rc, lh != null ? lh.getId() : -1);
}
if (checkAndCompleteLedgerOpTask(rc, lh, ctx)) {
return;
}
```
*Modification*
The timeout logic shouldn't modify the `ledgerCreated` context. It should let the callback itself to process
the `ledgerCreated` context.
* Change to use CAS
(cherry picked from commit a34f693)
|
Hi, |
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
* Fix producer stucks on creating ledger timeout
*Motivation*
The `ledgerCreated` flag is passed as ctx to the createLedger callback.
The callback already had the logic on handling `ledgerCreated` flag. But we set the flag to `false`
when timeout happens. It will trigger the following race condition:
a) The createComplete callback is triggered when timeout. But the pending add requests are not error'd out.
b) If the createComplete callback eventually completes, it will see `ledgerCreated` flag is set to true,
so it will cause `checkAndCompleteLedgerOpTask` returns false and exist too early without processing the
pending add requests.
This race condition only happens when creating ledger times out.
```
public synchronized void createComplete(int rc, final LedgerHandle lh, Object ctx) {
if (log.isDebugEnabled()) {
log.debug("[{}] createComplete rc={} ledger={}", name, rc, lh != null ? lh.getId() : -1);
}
if (checkAndCompleteLedgerOpTask(rc, lh, ctx)) {
return;
}
```
*Modification*
The timeout logic shouldn't modify the `ledgerCreated` context. It should let the callback itself to process
the `ledgerCreated` context.
* Change to use CAS
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The
ledgerCreatedflag is passed as ctx to the createLedger callback.The callback already had the logic on handling
ledgerCreatedflag. But we set the flag tofalsewhen timeout happens. It will trigger the following race condition:
a) The createComplete callback is triggered when timeout. But the pending add requests are not error'd out.
b) If the createComplete callback eventually completes, it will see
ledgerCreatedflag is set to true,so it will cause
checkAndCompleteLedgerOpTaskreturns false and exist too early without processing thepending add requests.
This race condition only happens when creating ledger times out.
Modification
The timeout logic shouldn't modify the
ledgerCreatedcontext. It should let the callback itself to processthe
ledgerCreatedcontext.