Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9878

Block AddPartitionsToTxn call until the txn markers are committed

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: core, producer
    • Labels:
      None

      Description

      Currently the EndTxn call from Producer will immediately return as the control record is written to the txn coordinator log. The ongoing transaction will be going to a pending state to wait for all txn markers to be propagated. In the meantime, producer client will start another new transaction but being rejected constantly until the pending state gets resolved, which is unnecessary round trips and more burden to the broker to handle repetitive requests.

      To avoid this situation, we should make the Producer client wait for txn marker completion instead. This will incur better performance overall, as no more back-off shall be triggered for a subsequent transaction to begin.

      On the other hand, we could also batch complete the AddPartitionsToTxn results if we buffered more than one request in the queue.

      The third change is on the client side, which is to maintain the futures of the AddPartitionsToTxn calls to make more inflight changes as necessary.

        Attachments

          Activity

            People

            • Assignee:
              zhaohaidao HaiyuanZhao
              Reporter:
              bchen225242 Boyang Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: