[KAFKA-9382] Allow consumers to commit offset in the middle of a rebalance

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: consumer
• Assignee: Jason Gustafson
• Reporter: Guozhang Wang
• Created: 07/Jan/20 23:52
• Updated: 07/Jan/20 23:52

Description

After we've done KAFKA-8421 we should consider letting consumers to allow committing in the middle of a rebalance — at the moment it will throw a non-fatal rebalance-in-progress exception — so that users do not need to worry and handle this transient error when unnecessary. It involves:

1) On client side, not checking the "REBALANCING" state and throw immediately.
2) On client side, capture and handle illegal generation if in "REBALANCING" and retry with the current assigned partitions.
3) On client side, use different connections for join/sync-group requests and for committing / fetching offsets.
4) On broker side, during CompletingRebalance accept the commit request with correct generation id instead of returning REBALANCE_IN_PROGRESS error code.