Details
Major Description
With message format v2, the log cleaner preserves the last offset in each batch even if the last record is removed. Currently when the batch is consumed by the consumer, we use the last record in the batch to determine the next offset to fetch. So if the last record in the batch was removed through compaction, the next fetch offset will still point to an offset in the current batch and it will be refetched. In the worst case, if the fetch size has room for that batch, the consumer will not be able to make progress. To fix this, we should advance the next fetch offset to the last offset from the batch once we have consumed that batch.
Attachments
Issue Links
- links to
