Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5865

Expiring batches with idempotence enabled could cause data loss.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0.0
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently we have a problem with this line:
      https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L282

      Because we can reset the producer id and return after draining batches, it means that we can drain batches for some partitions, then find a batch has expired, and then return. But the batches which were drained are now no longer in the producer queue, and haven't been sent. Thus they are totally lost, and the call backs will never be invoked.

      This is already fixed in https://github.com/apache/kafka/pull/3743 , but opening this in case we want to fix it in 0.11.0.2 as well.

        Attachments

          Activity

            People

            • Assignee:
              apurva Apurva Mehta
              Reporter:
              apurva Apurva Mehta
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: