Details
-
Improvement
-
Status: Open
-
Not a Priority
-
Resolution: Unresolved
-
1.11.1
-
None
Description
If I'll run multiple Flink instances with same consumer group id they will not re-balance partitions with each other, but rather each instance take all partitions, effectively not working in parallel at all, and multiplying amount of messages processed.
This is because FlinkKafkaConsumer has its own re-balancing mechanism for current parallelism level and then just calls:
`consumerTmp.assign(newPartitionAssignments);`
I suppose there has to be a way to fallback to default kafka mechanism of re-balancing to respect consumer group id, but it's not presented in Flink at all.