Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4815

Idempotent/transactional Producer (KIP-98)

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0.0
    • Component/s: clients, core, producer
    • Labels:

      Issue Links

      1.
      Implement idempotent producer Sub-task Resolved Apurva Mehta
       
      2.
      Message format changes for idempotent/transactional producer Sub-task Resolved Jason Gustafson
       
      3.
      Implement transactional clients Sub-task Resolved Apurva Mehta
       
      4.
      Add request/response classes for transactions (KIP-98) Sub-task Resolved Matthias J. Sax
       
      5.
      Consider disabling record level CRC checks for message format V2 Sub-task Resolved Jason Gustafson
       
      6.
      Consider making baseOffset the first offset in message format v2 Sub-task Resolved Jason Gustafson
       
      7.
      Exactly-once upgrade notes Sub-task Resolved Jason Gustafson
       
      8.
      Update protocol documentation to mention message format v2 Sub-task Resolved Apurva Mehta
       
      9.
      Update Message Delivery Semantics section to take into account KIP-98 Sub-task Resolved Jason Gustafson
       
      10.
      Old clients don't support message format V2 Sub-task Resolved Unassigned
       
      11.
      Additional validation in validateMessagesAndAssignOffsets Sub-task Resolved Unassigned
       
      12.
      Think through implications of max.message.size affecting record batches in message format V2 Sub-task Resolved Apurva Mehta
       
      13.
      Reconsider default retries for idempotent producer Sub-task Resolved Apurva Mehta
       
      14.
      Implement transaction index for KIP-98 Sub-task Resolved Jason Gustafson
       
      15.
      Avoid expensive initialization of producer state when upgrading Sub-task Resolved Jason Gustafson
       
      16.
      Add Integration tests for transactional producer Sub-task Resolved Apurva Mehta
       
      17.
      LogCleaner should be transaction-aware Sub-task Resolved Jason Gustafson
       
      18.
      KIP-98 : broker side handling for the TxnOffsetCommitRequest Sub-task Resolved Apurva Mehta
       
      19.
      Handle topic deletion for ongoing transactions Sub-task Resolved Guozhang Wang
       
      20.
      Remove retention time from TxnOffsetCommit RPC Sub-task Resolved Jason Gustafson
       
      21.
      Consumer GroupCoordinator should continue to materialize committed offsets in offset order even for transactional offset commits Sub-task Resolved Apurva Mehta
       
      22.
      TransactinoCoordinator does not bump epoch when aborting open transactions Sub-task Resolved Guozhang Wang
       
      23.
      Transaction index recovery does not snapshot properly Sub-task Resolved Jason Gustafson
       
      24.
      Producer should drop queued sends when transaction is aborted Sub-task Resolved Jason Gustafson
       
      25.
      Load only batch header when rebuilding producer ID map Sub-task Resolved Jason Gustafson
       
      26.
      Fix flaky test LogCleanerTest.testCommitMarkerRemoval Sub-task Resolved Jason Gustafson
       
      27.
      TransactionalId authorization should imply ProducerId authorization Sub-task Resolved Jason Gustafson
       
      28.
      Producer should not send AbortTxn unless transaction has actually begun Sub-task Resolved Damian Guy
       
      29.
      TransactionsBounceTest occasionally sees INVALID_TXN_STATE errors Sub-task Resolved Jason Gustafson
       
      30.
      TransactionBounceTest occasionally fails due to partition errors Sub-task Resolved Apurva Mehta
       
      31.
      Producer only needs to send AddOffsetsToTxn one time for each group Sub-task Resolved Unassigned
       
      32.
      KafkaConsumer.committed() should get latest committed offsets from the server Sub-task Resolved Apurva Mehta
       
      33.
      TransactionCoordinator must expire transactionalIds Sub-task Resolved Damian Guy
       
      34.
      Protect concurrent access to the cached transaction status Sub-task Resolved Guozhang Wang
       
      35.
      System tests for KIP-98 / transactions Sub-task Resolved Apurva Mehta
       
      36.
      Transactions integration test: Use factory methods to keep track of open producers and consumers and close them all on tearDown Sub-task Resolved Vahid Hashemian
       
      37.
      KafkaProducer's TransactionManager needs a review on synchronization Sub-task Resolved Apurva Mehta
       
      38.
      Update clients and server code to make sure that epoch and sequence numbers wrap around Sub-task Resolved Jason Gustafson
       
      39.
      TC should handle UNSUPPORTED_FOR_MESSAGE_FORMAT in WriteTxnMarker response Sub-task Resolved Damian Guy
       
      40.
      Update KIP-98 to reflect changes during implementation. Sub-task Resolved Apurva Mehta
       
      41.
      Resolve AddPartitions response error code inconsistency Sub-task Resolved Apurva Mehta
       
      42.
      Transactions system test with hard broker bounces fails sporadically Sub-task Resolved Apurva Mehta
       
      43.
      Add additional test cases for batch splitting to ensure idempotent/transactional metadata is preserved Sub-task Resolved Jason Gustafson
       
      44.
      Broker clean bounce test puts the broker into a 'CONCURRENT_TRANSACTIONS' state permanently Sub-task Resolved Apurva Mehta
       
      45.
      Broker returns messages beyond "latest stable offset" to transactional consumer in read_committed mode Sub-task Resolved Jason Gustafson
       
      46.
      StackOverFlow error in transaction coordinator Sub-task Resolved Damian Guy
       
      47.
      Producer attempts to send transactional messages before adding partitions to transaction Sub-task Resolved Apurva Mehta
       
      48.
      Fix regression in compressed message iteration affecting magic v0 and v1 Sub-task Resolved Jason Gustafson
       
      49.
      Last Stable Offset not returned in Fetch request Sub-task Resolved Jason Gustafson
       
      50.
      Transactions: Concurrent transactional consumer reads aborted messages Sub-task Resolved Jason Gustafson
       
      51.
      Transactional producer cannot find coordinator when trying to abort transaction after error Sub-task Resolved Jason Gustafson
       
      52.
      Transactional producer aborts batches incorrectly in abortable error state Sub-task Resolved Jason Gustafson
       
      53.
      Producer IllegalStateException: Batch has already been completed Sub-task Resolved Jason Gustafson
       
      54.
      Produce state lost if no snapshot retained Sub-task Resolved Jason Gustafson
       
      55.
      TransactionalMessageCopier should be force killed on test shutdown Sub-task Resolved Apurva Mehta
       
      56.
      NullPointerException when loading producer snapshot Sub-task Resolved Jason Gustafson
       
      57.
      UnsupportedOperationException in WriteTxnMarkers handler Sub-task Resolved Apurva Mehta
       
      58.
      Consumer should use last offset from batch to set next fetch offset Sub-task Resolved Jason Gustafson
       
      59.
      Deletion of tombstones during cleaning should consider idempotent message retention Sub-task Resolved Jason Gustafson
       

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user apurvam opened a pull request:

        https://github.com/apache/kafka/pull/2735

        KAFKA-4815 : Add idempotent producer semantics

        This is from the KIP-98 proposal.

        The main points of discussion surround the correctness logic, particularly the Log class where incoming entries are validated and duplicates are dropped, and also the producer error handling to ensure that the semantics are sound from the users point of view.

        There is some subtlety in the idempotent producer semantics. This patch only guarantees idempotent production upto the point where an error has to be returned to the user. Once we hit a such a non-recoverable error, we can no longer guarantee message ordering nor idempotence without additional logic at the application level.

        In particular, if an application wants guaranteed message order without duplicates, then it needs to do the following on the error callback:

        1. Close the producer so that no queued batches are sent. This is important for guaranteeing ordering.
        2. Read the tail of the log to inspect the last message committed. This is important for avoiding duplicates.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/confluentinc/kafka exactly-once-idempotent-producer

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/kafka/pull/2735.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #2735


        commit 83f5b2937d368efa768be6d7df1f57d99425a252
        Author: fpj <fpj@apache.org>
        Date: 2016-10-10T16:19:52Z

        Add broker producer id mapping support

        commit 37eac83364a67eebf0e1d96552d9fe864baa3d76
        Author: Guozhang Wang <wangguoz@gmail.com>
        Date: 2017-02-14T19:12:58Z

        KEOS: idempotent producer pid generation (#126)

        Add the transaction coordinator for pid generation and management.

        commit 6d918d0b44302aa81627f3f2748756356614f228
        Author: hachikuji <jason@confluent.io>
        Date: 2017-02-23T18:59:46Z

        Minor fixes for test failures and consistency (#133)

        commit d1868602be8b3e81d98644d673a73b9f730ad586
        Author: Jason Gustafson <jason@confluent.io>
        Date: 2017-03-01T22:40:30Z

        Fix headers, new checkstyle rules, and some breakage in KafkaApis

        commit f2c01a71508cd5c1eaf8c03a372f7dc308af3cf2
        Author: hachikuji <jason@confluent.io>
        Date: 2017-03-07T00:23:11Z

        Avoid removal of non-expired PIDs when log cleaning (#138)

        commit 31ff08eaf704b9e4fe6ee492277d981b47e8016c
        Author: Apurva Mehta <apurva.1618@gmail.com>
        Date: 2017-03-09T01:50:01Z

        Client side implementation of the idempotent producer. (#129)

        commit 439f284fd4efa56449d43f2fb5e14614689c7b80
        Author: Apurva Mehta <apurva@confluent.io>
        Date: 2017-03-09T21:23:55Z

        Fix build errors due to rebase

        commit 0a81a40258da29ed62caf745e7c1895ca7eead4c
        Author: hachikuji <jason@confluent.io>
        Date: 2017-03-10T17:50:01Z

        PIDs should be expired according to the transactional id expiration setting (#139)

        commit 2643e5b6bb6ad6b3d1cd6b15073a92396be5693b
        Author: Apurva Mehta <apurva@confluent.io>
        Date: 2017-03-15T18:45:07Z

        Fix build errors due to rebase

        commit 848137a28b02cd085605e92726d0457b0153e1c4
        Author: Apurva Mehta <apurva@confluent.io>
        Date: 2017-03-16T18:43:29Z

        Remove depependence of MemoryRecordsBuilder on TransactionState

        commit 15d23b65166cff6dbd7be0e925ff9ec811d7baac
        Author: hachikuji <jason@confluent.io>
        Date: 2017-03-23T22:30:39Z

        A few minor improvements (#150)

        commit fafffa8f2445e2a559f24e5286c948c616287f9a
        Author: Apurva Mehta <apurva.1618@gmail.com>
        Date: 2017-03-24T20:02:48Z

        Reset transaction state on all irrecoverable exceptions (#153)

        commit 637f864887e341c51f4fd7192016d2afc45bcf1e
        Author: Apurva Mehta <apurva@confluent.io>
        Date: 2017-03-24T21:08:29Z

        Fix build errors due to rebase


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2735 KAFKA-4815 : Add idempotent producer semantics This is from the KIP-98 proposal. The main points of discussion surround the correctness logic, particularly the Log class where incoming entries are validated and duplicates are dropped, and also the producer error handling to ensure that the semantics are sound from the users point of view. There is some subtlety in the idempotent producer semantics. This patch only guarantees idempotent production upto the point where an error has to be returned to the user. Once we hit a such a non-recoverable error, we can no longer guarantee message ordering nor idempotence without additional logic at the application level. In particular, if an application wants guaranteed message order without duplicates, then it needs to do the following on the error callback: Close the producer so that no queued batches are sent. This is important for guaranteeing ordering. Read the tail of the log to inspect the last message committed. This is important for avoiding duplicates. You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka exactly-once-idempotent-producer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2735.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2735 commit 83f5b2937d368efa768be6d7df1f57d99425a252 Author: fpj <fpj@apache.org> Date: 2016-10-10T16:19:52Z Add broker producer id mapping support commit 37eac83364a67eebf0e1d96552d9fe864baa3d76 Author: Guozhang Wang <wangguoz@gmail.com> Date: 2017-02-14T19:12:58Z KEOS: idempotent producer pid generation (#126) Add the transaction coordinator for pid generation and management. commit 6d918d0b44302aa81627f3f2748756356614f228 Author: hachikuji <jason@confluent.io> Date: 2017-02-23T18:59:46Z Minor fixes for test failures and consistency (#133) commit d1868602be8b3e81d98644d673a73b9f730ad586 Author: Jason Gustafson <jason@confluent.io> Date: 2017-03-01T22:40:30Z Fix headers, new checkstyle rules, and some breakage in KafkaApis commit f2c01a71508cd5c1eaf8c03a372f7dc308af3cf2 Author: hachikuji <jason@confluent.io> Date: 2017-03-07T00:23:11Z Avoid removal of non-expired PIDs when log cleaning (#138) commit 31ff08eaf704b9e4fe6ee492277d981b47e8016c Author: Apurva Mehta <apurva.1618@gmail.com> Date: 2017-03-09T01:50:01Z Client side implementation of the idempotent producer. (#129) commit 439f284fd4efa56449d43f2fb5e14614689c7b80 Author: Apurva Mehta <apurva@confluent.io> Date: 2017-03-09T21:23:55Z Fix build errors due to rebase commit 0a81a40258da29ed62caf745e7c1895ca7eead4c Author: hachikuji <jason@confluent.io> Date: 2017-03-10T17:50:01Z PIDs should be expired according to the transactional id expiration setting (#139) commit 2643e5b6bb6ad6b3d1cd6b15073a92396be5693b Author: Apurva Mehta <apurva@confluent.io> Date: 2017-03-15T18:45:07Z Fix build errors due to rebase commit 848137a28b02cd085605e92726d0457b0153e1c4 Author: Apurva Mehta <apurva@confluent.io> Date: 2017-03-16T18:43:29Z Remove depependence of MemoryRecordsBuilder on TransactionState commit 15d23b65166cff6dbd7be0e925ff9ec811d7baac Author: hachikuji <jason@confluent.io> Date: 2017-03-23T22:30:39Z A few minor improvements (#150) commit fafffa8f2445e2a559f24e5286c948c616287f9a Author: Apurva Mehta <apurva.1618@gmail.com> Date: 2017-03-24T20:02:48Z Reset transaction state on all irrecoverable exceptions (#153) commit 637f864887e341c51f4fd7192016d2afc45bcf1e Author: Apurva Mehta <apurva@confluent.io> Date: 2017-03-24T21:08:29Z Fix build errors due to rebase
        Hide
        ijuma Ismael Juma added a comment -

        I moved this JIRA to 0.11.1.0 as it includes a number of subtasks that won't be merged in 0.11.0.0. Ideally, we'd move the tasks that are not for 0.11.0.0 into another umbrella JIRA, but it's quite a bit of work and not clear if it's worth it.

        Show
        ijuma Ismael Juma added a comment - I moved this JIRA to 0.11.1.0 as it includes a number of subtasks that won't be merged in 0.11.0.0. Ideally, we'd move the tasks that are not for 0.11.0.0 into another umbrella JIRA, but it's quite a bit of work and not clear if it's worth it.

          People

          • Assignee:
            hachikuji Jason Gustafson
            Reporter:
            hachikuji Jason Gustafson
          • Votes:
            4 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development