[KAFKA-16758] Extend Consumer#close with option to leave the group or not

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: needs-kip
• Assignee: TengYao Chi
• Reporter: A. Sophie Blee-Goldman
• Created: 14/May/24 05:39
• Updated: 23/Sep/24 03:43

Description

See comments on https://issues.apache.org/jira/browse/KAFKA-16514 for the full context.

Essentially we would get rid of the "internal.leave.group.on.close" config that is used as a backdoor by Kafka Streams right now to prevent closed consumers from leaving the group, thus reducing unnecessary task movements after a simple bounce. 

This would be replaced by an actual public API that would allow the caller to opt in or out to the LeaveGroup when close is called. This would be similar to the KafkaStreams#close(CloseOptions) API, and in fact would be how that API will be implemented (since it only works for static groups at the moment as noted in KAFKA-16514 )

This has several benefits over the current situation:

1. It allows plain consumer apps to opt-out of leaving the group when closed, which is currently not possible through any public API (only an internal backdoor config)
2. It enables the caller to dynamically select the appropriate action depending on why the client is being closed – for example, you would not want the consumer to leave the group during a simple restart, but would want it to leave the group when shutting down the app or if scaling down the node. This is not possible today, even with the internal config, since configs are immutable
3. It can be leveraged to "fix" the KafkaStreams#close(closeOptions) API so that the user's choice to leave the group during close will be respected for non-static members

Issue Links

• blocks: KAFKA-16514 Kafka Streams: stream.close(CloseOptions) does not respect options.leaveGroup flag.
• mentioned in: Page Loading...