[KAFKA-8587] One producer per thread for Kafka Streams EOS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: streams
• Assignee: Boyang Chen
• Reporter: Boyang Chen
• Created: 23/Jun/19 03:30
• Updated: 05/May/20 05:55
• Resolved: 05/May/20 05:55

Description

Currently exactly-once producer is coupled with individual input partitions. This is not a well scaled solution for client application such as Kafka Streams, and the root cause is that EOS producer doesn't understand the topic partition move throughout consumer group rebalance. By amending this semantic gap, we could achieve much better EOS scalability.

Issue Links

• mentioned in: Page Loading...
• mentioned in: Page Loading...
• mentioned in: Page Loading...

Sub-Tasks

• Consumer fetch offset back-off with pending transactions: Resolved
• Add consumer group information to txn commit: Resolved
• Expose Consumer Group Metadata for Transactional Producer: Resolved
• Add examples for EOS standalone and group mode under kafka/examples: Resolved
• Integration test for new EOS model with vanilla Producer and Consumer: Resolved
• Add new sendOffsets API to include consumer group metadata: Resolved
• Refactor commit logic: Resolved
• Pass consumer group metadata to producer on commit: Resolved
• Add documentation for new stream EOS change: Resolved
• Replace transaction abortion scheduler with a delayed queue: Resolved
• Add configurable throw on unsupported protocol: Resolved
• Add System Test For ensuring the EOS-beta app would crash with broker downgrade: Resolved
• Use internal consumer config to fail for unsupported protocol: Resolved
• Producer could automatically downgrade TxnOffsetCommitRequest with internal flag: Resolved
• Extend EOS integration test for EOS-beta: Resolved
• Add EOS protocol changes to documentation: Resolved
• Shrink transaction timeout for Streams: Resolved
• Extend EOS system tests for EOS-beta: Resolved