[KAFKA-16235] auto commit still causes delays due to retriable UNKNOWN_TOPIC_OR_PARTITION

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0, 3.2.1
• Component/s: clients, consumer
• Assignee: Unassigned
• Reporter: Ryan Leslie
• Created: 07/Feb/24 19:01
• Updated: 08/Feb/24 18:00

Description

In KAFKA-12256 an issue was described where deleted topics can cause auto-commit to get stuck looping on UNKNOWN_TOPIC_OR_PARTITION, resulting in message delays. This had also been noted in KAFKA-13310 and a fix was made which was included in Kafka 3.2.0: https://github.com/apache/kafka/pull/11340

Unfortunately, that commit contributed to another more urgent issue, KAFKA-14024, and after subsequent code changes in https://github.com/apache/kafka/pull/12349, KAFKA-12256 was no longer fixed, and has been an issue again since 3.2.1+

This ticket is primarily for more visibility around this since KAFKA-12256 has been resolved for a long time now even though the issue exists. Ideally this behavior could once again be corrected in the existing consumer, but at this point most development effort appears to be focused on the next-gen consumer (KIP-848). I do see that for the next-gen consumer at least, these problems are being newly resurfaced and tracked in KAFKA-16233 and KAFKA-16224.

Issue Links

• relates to: KAFKA-16224 Fix handling of deleted topic when auto-committing before revocation
• relates to: KAFKA-14024 Consumer stuck during cooperative rebalance for Commit offset in onJoinPrepare
• relates to: KAFKA-13310 KafkaConsumer cannot jump out of the poll method, and the consumer is blocked in the ConsumerCoordinator method maybeAutoCommitOffsetsSync(Timer timer). Cpu and traffic of Broker's side increase sharply
• relates to: KAFKA-12256 auto commit causes delays due to retriable UNKNOWN_TOPIC_OR_PARTITION
• relates to: KAFKA-16233 Review auto-commit continuously committing when no progress