Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2590 KIP-28: Kafka Streams Checklist
  3. KAFKA-3193

Wait for producer message acks in StreamTask.commit()

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Today in StreamTask.commit(), we does the following:

      1) flush local state storage if it is persistent (e.g. RocksDB).
      2) flush produced records in the downstream and changelog Kafka topics.
      3) commit consumed offsets if it is dirty already

      However in step 2) above:

      producer.flush();
      

      does not actually block on all the acks, but only force draining all records in the buffer and send them. This actually result in a major defect of the commit() call. We'd better add the ability in producer to wait in flushing.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              guozhang Guozhang Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: