Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8021

KafkaProducer.flush() can show unexpected behavior when a batch is split

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.11.0.0
    • Fix Version/s: None
    • Component/s: clients, producer
    • Labels:
      None

      Description

      KafkaProducer.flush() marks the flush in progress and then waits for all incomplete batches to be completed (waits on the producer batch futures to finish).

      The behavior is seen when a batch is split due to MESSAGE_TOO_LARGE exception.

      The large batch is split into smaller batches (2 or more) but ProducerBatch.split() marks the large batch future as complete before adding the new batches in the incomplete list of batches. At this time if the KafkaProducer.flush() is called then it'll make a copy of existing incomplete list of batches and waits for them to complete while ignoring the large batch that was split into smaller batches.

       

        Attachments

          Activity

            People

            • Assignee:
              amendhekar Abhishek Mendhekar
              Reporter:
              amendhekar Abhishek Mendhekar
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: