[KAFKA-16933] New consumer leave group flow may not send leave request and clear epoch

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.7.0
• Fix Version/s: 3.8.0
• Component/s: clients, consumer
• Labels: kip-848-client-support
• Assignee: Lianet Magrans
• Reporter: Lianet Magrans
• Created: 11/Jun/24 14:39
• Updated: 15/Aug/24 13:55
• Resolved: 14/Jun/24 15:51

Description

When leaving a group (due to close or unsubscribe), the consumer needs to run revocation callbacks. In the case where this callbacks fail (user error or timeout due to long running callback), the consumer should still clear its epoch and send a leave group heartbeat. This is currently the case for unsubscribe but not for close, so should be fixed.

Also note that clearing the epoch even if the callback fails is needed to ensure that the last epoch is not used anymore in offset commit requests after the member leaves the group (ex. when auto-commit enabled).  

Also note that this should still be applied (clear epoch) if a member that is running the callbacks to leave get fenced. It does not need to send the leave group request but should take all leaving action to leave the client in a consistent state (avoiding auto-commits that could happen while it leaves, that would use the fenced epoch) 

Issue Links

• fixes: KAFKA-16022 AsyncKafkaConsumer sometimes complains “No current assignment for partition {}”
• is related to: KAFKA-16290 Investigate propagating subscription state updates via queues
• links to: GitHub Pull Request #16272