Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4815 Idempotent/transactional Producer (KIP-98)
  3. KAFKA-5317

Update KIP-98 to reflect changes during implementation.




      While implementing the EOS design, there are some minor (or major?) tweaks we made as we hands-on the code bases. We will compile all these changes in a single run at the end of the code-ready and this JIRA is for keeping track of all the changes we made.

      02/27/2017: collapse the two types of transactional log messages into a single type with key as transactional id, and value as [pid, epoch, transaction_timeout, transaction_state, [topic partition ] ]. Also using a single memory map in cache instead of two on the TC.

      03/01/2017: for pid expiration, we decided to use min(transactional id expiration timeout, topic retention). For topics enabled for compaction only, we just use the transactional timeout. If the retention setting is larger than the transactional id expiration timeout, then the pid will be "logically" expired (i.e. we will remove it from the cached pid mapping and ignore it when rebuilding the cache from the log)

      03/20/2017: add a new exception type in `o.a.k.common.errors` for invalid transaction timeout values.

      03/25/2017: extend WriteTxnMarkerRequest to contain multiple markers for multiple PIDs with a single request.

      04/20/2017: add transactionStartTime to TransactionMetadata

      04/26/2017: added a new retriable error: Errors.CONCURRENT_TRANSACTIONS

      04/01/2017: We also enforce acks=all on the client when idempotence is enabled. Without this, we cannot again guarantee idemptoence.

      04/10/2017: WE also don't pass the underlying exception to `RetriableOffsetCommitFailedException` anymore: https://issues.apache.org/jira/browse/KAFKA-5052




            apurva Apurva Mehta
            apurva Apurva Mehta
            0 Vote for this issue
            3 Start watching this issue