[KAFKA-17064] New consumer assign should update assignment in background thread

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.8.0
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: consumer-threading-refactor kip-848-client-support
• Assignee: PoAn Yang
• Reporter: Lianet Magrans
• Created: 02/Jul/24 14:35
• Updated: 30/Aug/24 11:25
• Resolved: 30/Aug/24 00:57

Description

With the new async consumer, the subscriptionState object is shared between the app thread and the background thread, but in principle all updates to the assignment should happen in the background thread, to avoid race conditions. Note that it's in the background where most updates to the assignment happen, as result of app event processing like unsubscribe, reconciliations, etc.). 

We've faced such races in places like unsubscribe and close, fixed by ensuring that all assignment updates happen in the background, and this also needs to be reviewed for the consumer.assign. The current implementation triggers an AssignmentChange event that is processed in the background, but that event is not really changing the assignment. It only commits offsets, and the assignment is updated in the app thread by calling subscriptionState.assignFromUser 

We should consider moving the assignment update to the background thread, as part of the AssignmentChangeEvent.  

Issue Links

• fixes: KAFKA-16022 AsyncKafkaConsumer sometimes complains “No current assignment for partition {}”
• is related to: KAFKA-16290 Investigate propagating subscription state updates via queues
• relates to: KAFKA-17448 New consumer seek should update positions in background thread
• links to: GitHub Pull Request #16673