Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4815 Idempotent/transactional Producer (KIP-98)
  3. KAFKA-5443

Consumer should use last offset from batch to set next fetch offset

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0.0
    • Component/s: clients, core, producer
    • Labels:
      None

      Description

      With message format v2, the log cleaner preserves the last offset in each batch even if the last record is removed. Currently when the batch is consumed by the consumer, we use the last record in the batch to determine the next offset to fetch. So if the last record in the batch was removed through compaction, the next fetch offset will still point to an offset in the current batch and it will be refetched. In the worst case, if the fetch size has room for that batch, the consumer will not be able to make progress. To fix this, we should advance the next fetch offset to the last offset from the batch once we have consumed that batch.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hachikuji Jason Gustafson
                Reporter:
                hachikuji Jason Gustafson
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: