Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4815

Idempotent/transactional Producer (KIP-98)

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0.0
    • Component/s: clients, core, producer
    • Labels:

      Attachments

        Issue Links

        1.
        Implement idempotent producer Sub-task Resolved Apurva Mehta
        2.
        Message format changes for idempotent/transactional producer Sub-task Resolved Jason Gustafson
        3.
        Implement transactional clients Sub-task Resolved Apurva Mehta
        4.
        Add request/response classes for transactions (KIP-98) Sub-task Resolved Matthias J. Sax
        5.
        Consider disabling record level CRC checks for message format V2 Sub-task Resolved Jason Gustafson
        6.
        Consider making baseOffset the first offset in message format v2 Sub-task Resolved Jason Gustafson
        7.
        Exactly-once upgrade notes Sub-task Resolved Jason Gustafson
        8.
        Update protocol documentation to mention message format v2 Sub-task Resolved Apurva Mehta
        9.
        Update Message Delivery Semantics section to take into account KIP-98 Sub-task Resolved Jason Gustafson
        10.
        Old clients don't support message format V2 Sub-task Resolved Unassigned
        11.
        Additional validation in validateMessagesAndAssignOffsets Sub-task Resolved Unassigned
        12.
        Think through implications of max.message.size affecting record batches in message format V2 Sub-task Resolved Apurva Mehta
        13.
        Reconsider default retries for idempotent producer Sub-task Resolved Apurva Mehta
        14.
        Implement transaction index for KIP-98 Sub-task Resolved Jason Gustafson
        15.
        Avoid expensive initialization of producer state when upgrading Sub-task Resolved Jason Gustafson
        16.
        Add Integration tests for transactional producer Sub-task Resolved Apurva Mehta
        17.
        LogCleaner should be transaction-aware Sub-task Resolved Jason Gustafson
        18.
        KIP-98 : broker side handling for the TxnOffsetCommitRequest Sub-task Resolved Apurva Mehta
        19.
        Handle topic deletion for ongoing transactions Sub-task Resolved Guozhang Wang
        20.
        Remove retention time from TxnOffsetCommit RPC Sub-task Resolved Jason Gustafson
        21.
        Consumer GroupCoordinator should continue to materialize committed offsets in offset order even for transactional offset commits Sub-task Resolved Apurva Mehta
        22.
        TransactinoCoordinator does not bump epoch when aborting open transactions Sub-task Resolved Guozhang Wang
        23.
        Transaction index recovery does not snapshot properly Sub-task Resolved Jason Gustafson
        24.
        Producer should drop queued sends when transaction is aborted Sub-task Resolved Jason Gustafson
        25.
        Load only batch header when rebuilding producer ID map Sub-task Resolved Jason Gustafson
        26.
        Fix flaky test LogCleanerTest.testCommitMarkerRemoval Sub-task Resolved Jason Gustafson
        27.
        TransactionalId authorization should imply ProducerId authorization Sub-task Resolved Jason Gustafson
        28.
        Producer should not send AbortTxn unless transaction has actually begun Sub-task Resolved Damian Guy
        29.
        TransactionsBounceTest occasionally sees INVALID_TXN_STATE errors Sub-task Resolved Jason Gustafson
        30.
        TransactionBounceTest occasionally fails due to partition errors Sub-task Resolved Apurva Mehta
        31.
        Producer only needs to send AddOffsetsToTxn one time for each group Sub-task Resolved Unassigned
        32.
        KafkaConsumer.committed() should get latest committed offsets from the server Sub-task Resolved Apurva Mehta
        33.
        TransactionCoordinator must expire transactionalIds Sub-task Resolved Damian Guy
        34.
        Protect concurrent access to the cached transaction status Sub-task Resolved Guozhang Wang
        35.
        System tests for KIP-98 / transactions Sub-task Resolved Apurva Mehta
        36.
        Transactions integration test: Use factory methods to keep track of open producers and consumers and close them all on tearDown Sub-task Resolved Vahid Hashemian
        37.
        KafkaProducer's TransactionManager needs a review on synchronization Sub-task Resolved Apurva Mehta
        38.
        Update clients and server code to make sure that epoch and sequence numbers wrap around Sub-task Resolved Jason Gustafson
        39.
        TC should handle UNSUPPORTED_FOR_MESSAGE_FORMAT in WriteTxnMarker response Sub-task Resolved Damian Guy
        40.
        Update KIP-98 to reflect changes during implementation. Sub-task Resolved Apurva Mehta
        41.
        Resolve AddPartitions response error code inconsistency Sub-task Resolved Apurva Mehta
        42.
        Transactions system test with hard broker bounces fails sporadically Sub-task Resolved Apurva Mehta
        43.
        Add additional test cases for batch splitting to ensure idempotent/transactional metadata is preserved Sub-task Resolved Jason Gustafson
        44.
        Broker clean bounce test puts the broker into a 'CONCURRENT_TRANSACTIONS' state permanently Sub-task Resolved Apurva Mehta
        45.
        Broker returns messages beyond "latest stable offset" to transactional consumer in read_committed mode Sub-task Resolved Jason Gustafson
        46.
        StackOverFlow error in transaction coordinator Sub-task Resolved Damian Guy
        47.
        Producer attempts to send transactional messages before adding partitions to transaction Sub-task Resolved Apurva Mehta
        48.
        Fix regression in compressed message iteration affecting magic v0 and v1 Sub-task Resolved Jason Gustafson
        49.
        Last Stable Offset not returned in Fetch request Sub-task Resolved Jason Gustafson
        50.
        Transactions: Concurrent transactional consumer reads aborted messages Sub-task Resolved Jason Gustafson
        51.
        Transactional producer cannot find coordinator when trying to abort transaction after error Sub-task Resolved Jason Gustafson
        52.
        Transactional producer aborts batches incorrectly in abortable error state Sub-task Resolved Jason Gustafson
        53.
        Producer IllegalStateException: Batch has already been completed Sub-task Resolved Jason Gustafson
        54.
        Produce state lost if no snapshot retained Sub-task Resolved Jason Gustafson
        55.
        TransactionalMessageCopier should be force killed on test shutdown Sub-task Resolved Apurva Mehta
        56.
        NullPointerException when loading producer snapshot Sub-task Resolved Jason Gustafson
        57.
        UnsupportedOperationException in WriteTxnMarkers handler Sub-task Resolved Apurva Mehta
        58.
        Consumer should use last offset from batch to set next fetch offset Sub-task Resolved Jason Gustafson
        59.
        Deletion of tombstones during cleaning should consider idempotent message retention Sub-task Resolved Jason Gustafson

          Activity

            People

            • Assignee:
              hachikuji Jason Gustafson
              Reporter:
              hachikuji Jason Gustafson
            • Votes:
              4 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: