[KAFKA-4815] Idempotent/transactional Producer (KIP-98)

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0.0
• Component/s: clients, core, producer
• Labels: kip
• Assignee: Jason Gustafson
• Reporter: Jason Gustafson
• Created: 28/Feb/17 20:54
• Updated: 30/May/19 08:16
• Resolved: 27/Jun/17 14:21

Description

This issue tracks implementation progress for KIP-98: https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging.

Issue Links

• links to: GitHub Pull Request #2735
• mentioned in: Page Loading...

Sub-Tasks

• Implement idempotent producer: Resolved
• Message format changes for idempotent/transactional producer: Resolved
• Implement transactional clients: Resolved
• Add request/response classes for transactions (KIP-98): Resolved
• Consider disabling record level CRC checks for message format V2: Resolved
• Consider making baseOffset the first offset in message format v2: Resolved
• Exactly-once upgrade notes: Resolved
• Update protocol documentation to mention message format v2: Resolved
• Update Message Delivery Semantics section to take into account KIP-98: Resolved
• Old clients don't support message format V2: Resolved
• Additional validation in validateMessagesAndAssignOffsets: Resolved
• Think through implications of max.message.size affecting record batches in message format V2: Resolved
• Reconsider default retries for idempotent producer: Resolved
• Implement transaction index for KIP-98: Resolved
• Avoid expensive initialization of producer state when upgrading: Resolved
• Add Integration tests for transactional producer: Resolved
• LogCleaner should be transaction-aware: Resolved
• KIP-98 : broker side handling for the TxnOffsetCommitRequest: Resolved
• Handle topic deletion for ongoing transactions: Resolved
• Remove retention time from TxnOffsetCommit RPC: Resolved
• Consumer GroupCoordinator should continue to materialize committed offsets in offset order even for transactional offset commits: Resolved
• TransactinoCoordinator does not bump epoch when aborting open transactions: Resolved
• Transaction index recovery does not snapshot properly: Resolved
• Producer should drop queued sends when transaction is aborted: Resolved
• Load only batch header when rebuilding producer ID map: Resolved
• Fix flaky test LogCleanerTest.testCommitMarkerRemoval: Resolved
• TransactionalId authorization should imply ProducerId authorization: Resolved
• Producer should not send AbortTxn unless transaction has actually begun: Resolved
• TransactionsBounceTest occasionally sees INVALID_TXN_STATE errors: Resolved
• TransactionBounceTest occasionally fails due to partition errors: Resolved
• Producer only needs to send AddOffsetsToTxn one time for each group: Resolved
• KafkaConsumer.committed() should get latest committed offsets from the server: Resolved
• TransactionCoordinator must expire transactionalIds: Resolved
• Protect concurrent access to the cached transaction status: Resolved
• System tests for KIP-98 / transactions: Resolved
• Transactions integration test: Use factory methods to keep track of open producers and consumers and close them all on tearDown: Resolved
• KafkaProducer's TransactionManager needs a review on synchronization: Resolved
• Update clients and server code to make sure that epoch and sequence numbers wrap around: Resolved
• TC should handle UNSUPPORTED_FOR_MESSAGE_FORMAT in WriteTxnMarker response: Resolved
• Update KIP-98 to reflect changes during implementation.: Resolved
• Resolve AddPartitions response error code inconsistency: Resolved
• Transactions system test with hard broker bounces fails sporadically: Resolved
• Add additional test cases for batch splitting to ensure idempotent/transactional metadata is preserved: Resolved
• Broker clean bounce test puts the broker into a 'CONCURRENT_TRANSACTIONS' state permanently: Resolved
• Broker returns messages beyond "latest stable offset" to transactional consumer in read_committed mode: Resolved
• StackOverFlow error in transaction coordinator: Resolved
• Producer attempts to send transactional messages before adding partitions to transaction: Resolved
• Fix regression in compressed message iteration affecting magic v0 and v1: Resolved
• Last Stable Offset not returned in Fetch request: Resolved
• Transactions: Concurrent transactional consumer reads aborted messages: Resolved
• Transactional producer cannot find coordinator when trying to abort transaction after error: Resolved
• Transactional producer aborts batches incorrectly in abortable error state: Resolved
• Producer IllegalStateException: Batch has already been completed: Resolved
• Produce state lost if no snapshot retained: Resolved
• TransactionalMessageCopier should be force killed on test shutdown: Resolved
• NullPointerException when loading producer snapshot: Resolved
• UnsupportedOperationException in WriteTxnMarkers handler: Resolved
• Consumer should use last offset from batch to set next fetch offset: Resolved
• Deletion of tombstones during cleaning should consider idempotent message retention: Resolved