[KAFKA-17726] New consumer subscribe/subscribeFromPattern in background thread

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: consumer-threading-refactor kip-848-client-support
• Assignee: PoAn Yang
• Reporter: Lianet Magrans
• Created: 08/Oct/24 14:16
• Updated: Yesterday 15:42

Description

There are still some calls to change the subscriptionState subscription in the app thread, that could lead to race-conditions given that the subscription state object is shared between the background and app thread and is mostly updated in the background. We should consider moving the subscriptionState.subscribe() and subscribeFromPattern calls to the background.

https://github.com/apache/kafka/blob/5624bc7c7e43339eb64e18491ac5039324e50678/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1465

https://github.com/apache/kafka/blob/5624bc7c7e43339eb64e18491ac5039324e50678/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1703

https://github.com/apache/kafka/blob/5624bc7c7e43339eb64e18491ac5039324e50678/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1736

One option would be to use the existing SubscriptionChangeEvent, perform the subscription (and metadata) changes in the AppEventProcessor (where unsubscribe already takes place). 

Note that these calls are closely related to metadata object calls, to request metadata for new topics and "maybeUpdateSubscriptionMetadata". We should maybe consider moving them together to the background (where the metadata object is also updated).  https://github.com/apache/kafka/blob/5624bc7c7e43339eb64e18491ac5039324e50678/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1703-L1705 

Issue Links

• links to: GitHub Pull Request #17569