Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9878

Block AddPartitionsToTxn call until the txn markers are committed

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • core, producer
    • None

    Description

      Currently the EndTxn call from Producer will immediately return as the control record is written to the txn coordinator log. The ongoing transaction will be going to a pending state to wait for all txn markers to be propagated. In the meantime, producer client will start another new transaction but being rejected constantly until the pending state gets resolved, which is unnecessary round trips and more burden to the broker to handle repetitive requests.

      To avoid this situation, we should make the Producer client wait for txn marker completion instead. This will incur better performance overall, as no more back-off shall be triggered for a subsequent transaction to begin.

      On the other hand, we could also batch complete the AddPartitionsToTxn results if we buffered more than one request in the queue.

      The third change is on the client side, which is to maintain the futures of the AddPartitionsToTxn calls to make more inflight changes as necessary.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zhaohaidao HaiyuanZhao
            bchen225242 Boyang Chen

            Dates

              Created:
              Updated:

              Slack

                Issue deployment