[KAFKA-16290] Investigate propagating subscription state updates via queues

• Type: Task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: consumer-threading-refactor kip-848 kip-848-client-support
• Assignee: Lianet Magrans
• Reporter: Lucas Brutschy
• Created: 21/Feb/24 09:07
• Updated: 15/Aug/24 13:58
• Resolved: 15/Aug/24 13:47

Description

We are mostly using the queues for interaction between application thread and background thread, but the subscription object is shared between the threads, and it is updated directly without going through the queues.

The way we allow updates to the subscription state from both threads is definitely not right, and will bring trouble. Places like the assign() is probably the most obvious, where we send an event to the background to commit, but then update the subscription in the foreground right away.

It seems sensible to aim for having all updates to the subscription state in the background, triggered from the app thread via events (and I think we already have related events for all updates, just that the subscription state was left out in the app thread).

Issue Links

• relates to: KAFKA-16227 Console consumer fails with `IllegalStateException`
• relates to: KAFKA-16194 KafkaConsumer.groupMetadata() should be correct when first records are returned
• relates to: KAFKA-16933 New consumer leave group flow may not send leave request and clear epoch
• relates to: KAFKA-16954 Move consumer leave operations on close to background thread
• relates to: KAFKA-17064 New consumer assign should update assignment in background thread
• relates to: KAFKA-17066 New consumer updateFetchPositions should perform all operations in background thread
• relates to: KAFKA-16315 Investigate propagating metadata updates via queues