[KAFKA-17116] New consumer may not send effective leave group if member ID received after close

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.8.0
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: kip-848-client-support needs-kip
• Assignee: TengYao Chi
• Reporter: Lianet Magrans
• Created: 10/Jul/24 21:05
• Updated: 5 days ago 08:52

Description

If the new consumer is closed after sending a HB to join, but before receiving the response to it, it will send a leave group request but without member ID (will simply fail with UNKNOWN_MEMBER_ID). This will make that the broker will have a registered new member, for which it will never receive a leave request for it.

1. consumer.subscribe -> sends HB to join, transitions to JOINING
2. consumer.close -> will transition to LEAVING and send HB with epoch -1 (without waiting for in-flight requests)
3. consumer receives response to initial HB, containing the assigned member ID. It will simply ignore it because it's not in the group anymore (UNSUBSCRIBED)

Note that the expectation, with the current logic, and main downsides of this are:

1. If the case was that the member received partitions on the first HB, those partitions won't be re-assigned (broker waiting for the closed consumer to reconcile them), until the rebalance timeout expires. 
2. Even if no partitions were assigned to it, the member will remain in the group from the broker point of view (but not from the client POV). The member will be eventually kicked out for not sending HBs, but only when it's session timeout expires.

Issue Links

• relates to: KAFKA-17115 Closing newly-created legacy consumers during rebalance can cause rebalances to hang
• links to: GitHub Pull Request #16649
• mentioned in: Page Loading...