[KAFKA-16285] Make group metadata available when a new assignment is set in async Kafka consumer

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.8.0
• Component/s: clients, consumer
• Labels: kip-848-client-support
• Assignee: Bruno Cadonna
• Reporter: Bruno Cadonna
• Created: 20/Feb/24 12:36
• Updated: 11/Mar/24 08:16
• Resolved: 04/Mar/24 11:43

Description

Currently, the new async Kafka consumer sends an event from the background thread to the application thread when the group metadata is updated. Group metadata is updated when the background thread receives a new assignment. More specifically, the member epoch is updated each time a new assignment is received and and the member ID is updated with the first assignment.
In contrast to the group metadata update, the assignment is directly set in the subscription without sending an update event from the background thread to the application thread. That means that there is a delay between the application thread being aware of the update to the assignment and the application thread being aware of the update to the group metadata. This behavior differs with respect to the legacy consumer were the assignment and the group metadata is updated at the same time.
We should make the update to the group metadata available to the application thread when the update to the assignment is made available to the application thread so that assignment an group metadata are in sync.

For example, producer.sendOffsetsToTransaction(offsetsToCommit, groupMetadata); benefits from this improvement because if the offsets to commit are consistent with the current assignment also the group metadata would be. Currently, that is not guaranteed.

Issue Links

• links to: GitHub Pull Request #15426