Test build #89177 has finished for PR 21038 at commit f317dec.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

@jerryshao Thank you very much for this issue.
I go to compile and test.

@koeninger would you please help to review, thanks!

The log in the jira looks like it's from a consumer rebalance, i.e. more than one driver consumer was running with the same group id.

Isn't the underlying problem here that the user is creating multiple streams with the same group id, despite what the documentation says? The log even says s/he copy-pasted the documentation group id "group use_a_separate_group_id_for_each_stream"

I don't think we should silently "fix" that. As a user, I wouldn't expect app A to suddenly start processing only half of the partitions just because entirely different app B started with the (misconfigured) same group id.

Thanks @koeninger for your comments. I think your suggestion is valid, the log here is just pasted from JIRA, but we also got the same issue from customer's report.

Here in the PR description, I mentioned that using two apps with same group id to mimic this issue. But I'm not sure the real use case from our customer, maybe in their scenario such usage is valid.

So I'm wondering if we can add a configuration to control whether it should be fail or just warning. Also I think exception/warning log should be improved to directly tell user about consumer rebalance issue, rather than throwing from Kafka as "no current assignment for partition xxx".

Thanks @koeninger , then I will just improve the exception message.

Test build #89230 has finished for PR 21038 at commit a7770e9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

Ping @koeninger , would you please help to review again. Thanks!

Seems like that should help address the confusion. Merging to master.

Thanks @koeninger for the review.

I think "This is fundamentally not correct, different apps should use different consumer group" statement is wrong. .According to Kafka Having consumers as part of the same consumer group means providing the “competing consumers” pattern with whom the messages from topic partitions are spread across the members of the group.
It seams Spark-Fafka cannot support this for now

@SehanRathnayake Kafka is designed for at most one consumer per partition per consumer group at any given point in time, https://kafka.apache.org/documentation/#design_consumerposition
Spark already manages creating a consumer per partition for the consumer group associated with a stream.
If you have a valid use case for running multiple Spark applications with the same consumer group, please explain it in a jira, not discussion of a pull request that has already been merged.

@koeninger According to Kafka documentation

If all the consumer instances have the same consumer group, then the records will effectively be load-balanced over the consumer instances
This means I can have multiple consumers with the same groupId which can help me to load balance my application and scale accordingly.
I don't know why it is said "fundamentally wrong" to have multiple consumers with the same groupId in spark.
So how can I achieve scalability to listen to a single partition and increase consumption rate with multiple spark consumers?
Is this the spark design fault or any other way to achieve scaling which I am unaware of?

@SehanRathnayake Any thoughts?

Hey @koeninger thanks for the reply
The issue with my application is I have one topic with 3 partitions once I start my application (Spark consumer) it listens to all the 3 partitions
LOG found as below
Setting newly assigned partitions [topic.partition-2, topic.partition-1, topic.partition-0]

When I start another instance of the same application with the same group id I can see there is rebalance in spark and one partition is assigned to the second application instance

LOG as follows in the first application instance

Setting newly assigned partitions [topic.partition-2, topic.partition-0]

So we can see the topic.partition 1 is assigned to the second instance of an application in the rebalancing process
But just after the above-mentioned log there is an exception as follows

java.lang.IllegalStateException: Previously tracked partitions [topic.partition-1] been revoked by Kafka because of consumer rebalance. This is mostly due to another stream with same group id joined, please check if there're different streaming application misconfigure to use the same group id. Fundamentally different stream should use different group id

And the application exits.
How can I have multiple consumers with same groupId for different partitions?
I also provided Assingmnt strategy as RoundRobin
kafkaParam.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.RoundRobinAssignor");

@koeninger Hi， we have prepared two spark streaming applications with the same group id to run respectively on different cluster for disaster recovery，the first application will failed when the second application started a few minutes later, and threw exception as:
java.lang.IllegalStateException: No current assignment for partition xxx

Why can't you use a different group id?

Why can't you use a different group id?

If the two spark streaming applications use different group id, the data will be processed twice and the result (in hbase) will be wrong

You already have to handle data being processed twice, or you're getting bad results in the event of a failure.