[KAFKA-15615] Improve handling of fetching during metadata updates

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.8.0
• Component/s: clients, consumer
• Labels: consumer-threading-refactor fetcher
• Assignee: appchemist
• Reporter: Kirk True
• Created: 16/Oct/23 22:29
• Updated: 07/Oct/24 04:01

Description

During a review of the new fetcher, junrao found what appears to be an opportunity for optimization.

When a fetch response receives an error about partition leadership, fencing, etc. a metadata refresh is triggered. However, it takes time for that refresh to occur, and in the interim, it appears that the consumer will blindly attempt to fetch data for the partition again, in kind of a "definition of insanity" type of way. Ideally, the consumer would have a way to temporarily ignore those partitions, in a way somewhat like the "pausing" approach so that they are skipped until the metadata refresh response is fully processed.

This affects both the existing KafkaConsumer and the new PrototypeAsyncConsumer.

Issue Links

• links to: GitHub Pull Request #15647