Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9860

Transactional Producer could pre-add partitions

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • producer

    Description

      As of today, the Producer transaction manager bookkeeps the partitions involved with current transaction. Each time it sees a new partition, it will try to send a request to add all the involved partitions to the broker, which results in multiple requests. If we could batch the work at the beginning of the transaction, we save unnecessary round trips.

      The idea is that most times the output partitions for a Producer is constant overtime, so we could leverage the last transactions affected partitions to do a batch `AddPartitionToTxn` first, and bump the EndTxn request with a field of partitions actually being added. The transaction coordinator will only send markers to the partitions included in the EndTxn. If the first batch is not a superset of affected partitions as we are producing data, we would still need a second AddPartition call. 

      The cons for this approach is unnecessary marker requests will be sent out when an ongoing transaction never gets the EndTxn call and has to be expired by the broker. The impact should be minimal the affected scope is always defined by the previous transaction.

      Attachments

        Activity

          People

            Unassigned Unassigned
            bchen225242 Boyang Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: