-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix deadlock in DruidStatement & DruidConnection #6868
Conversation
@@ -63,7 +63,7 @@ public DruidConnection(final String connectionId, final int maxStatements, final | |||
this.connectionId = Preconditions.checkNotNull(connectionId); | |||
this.maxStatements = maxStatements; | |||
this.context = ImmutableMap.copyOf(context); | |||
this.statements = new HashMap<>(); | |||
this.statements = new ConcurrentHashMap<>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@zhaojiandong For clarity's sake, could you please change the type of the field to ConcurrentMap, & a comment about why it's both @GuardedBy
and Concurrent?
// ConcurrentMap even though it is usually accessed synchronized, because it's unsynchronized
// in one case: the onClose function passed into DruidStatements contained by the map. This is
// done to avoid deadlocks; see https://github.com/apache/incubator-druid/issues/6867.
@GuardedBy("statements")
private final ConcurrentMap<Integer, DruidStatement> statements;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, I'll put some comments on statements and onClose, thank for review
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@zhaojiandong For clarity's sake, could you please change the type of the field to ConcurrentMap, & a comment about why it's both
@GuardedBy
and Concurrent?// ConcurrentMap even though it is usually accessed synchronized, because it's unsynchronized // in one case: the onClose function passed into DruidStatements contained by the map. This is // done to avoid deadlocks; see https://github.com/apache/incubator-druid/issues/6867. @GuardedBy("statements") private final ConcurrentMap<Integer, DruidStatement> statements;
How about use another sync flag: connectionLock, and the type of statements is ConcurrentHashMap ?
In this way GuardedBy check will be passed. @gianm
sql/src/main/java/org/apache/druid/sql/avatica/DruidStatement.java
Outdated
Show resolved
Hide resolved
Hi @zhaojiandong, I think something went wrong with a merge in this patch. It has all of a sudden started involving a lot of unrelated code. |
Sorry, I've merged some commit from master by mistake @gianm |
a623f05
to
4cbd40e
Compare
Hi @gianm , I force to push a clean branch to this pr |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good... I had one last suggestion about a clarifying comment & type change. Let me know what you think.
sql/src/main/java/org/apache/druid/sql/avatica/DruidConnection.java
Outdated
Show resolved
Hide resolved
sql/src/main/java/org/apache/druid/sql/avatica/DruidConnection.java
Outdated
Show resolved
Hide resolved
4cbd40e
to
899869d
Compare
|
||
@GuardedBy("statements") | ||
@GuardedBy("connectionLock") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need to add @GuardedBy
annotation to connectionLock
itself here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @asdf2014 , the field connectionLock now needs to be accessed synchronized, So I think should add an annotation here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmm... Please help me figure out that our initial goal should be to hope that statements
can be accessed synchronously. Why not add the @GuardedBy("connectionLock")
annotation to the statements
property, but add the connectionLock
object itself. So I don't understand what this annotation means here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because we use field connectionLock as a synchronized flag, connectionLock now protect all methods. we do not need connectionLock to access statements, since the type of statements is ConcurrentMap.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To me, it makes sense because statements
is now not always guarded by connectionLock
. But, it is in some (well, many) cases. The comment explains it in a way that the annotation would struggle to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for explaining this @gianm @zhaojiandong . I mean we shouldn't remove the @GuardedBy("connectionLock")
from statements
because the statements
is not always synchronized by connectionLock
and the CI fails, which is unreasonable. In addition, putting a @GuardedBy
on connectionLock
is also not necessary. If this is done, it means that we should have the connectionLock
lock object when we use connectionLock
, in fact it cannot achieve the purpose that we should own the connectionLock
when using statements
. So, i think there is a better way to solve this situation by adding a @SuppressWarnings("FieldAccessNotGuarded")
annotation to the Runnable#run
method, or adding a // noinspection FieldAccessNotGuarded
comment to the statements.remove(statementId)
caluse, which will suppress this check and make CI happy.
@GuardedBy("connectionLock")
private final ConcurrentMap<Integer, DruidStatement> statements;
private final Object connectionLock = new Object();
public DruidStatement createStatement(SqlLifecycleFactory sqlLifecycleFactory)
{
// ...
final DruidStatement statement = new DruidStatement(
// ...
() -> {
// noinspection FieldAccessNotGuarded
statements.remove(statementId);
}
);
// ...
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I raised up a PR #6903, PTAL. @gianm @zhaojiandong
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gianm You are welcome, I am sorry for my poor English expression 😰
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks @zhaojiandong
* Fix deadlock in DruidStatement & DruidConnection * change statements type to ConcurrentMap
* Fix deadlock in DruidStatement & DruidConnection * change statements type to ConcurrentMap
Fix issue #6867
We found deadlock in DruidStatement and DruidConnection, this will cause exhaust connections.
The statement thread will hold the lock, but when the slow query occurs in statement. The connection close signal from client or server, will cause deadlock with statement thread.