[KAFKA-12177] Retention is not idempotent

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Assignee: Lucas Bradstreet
• Reporter: Lucas Bradstreet
• Created: 11/Jan/21 19:13
• Updated: 02/Mar/21 21:12
• Resolved: 02/Mar/21 21:12

Description

Kafka today applies retention in the following order:

1. Time
2. Size
3. Log start offset

Today it is possible for a segment with offsets less than the log start offset to contain data that is not deletable due to time retention. This means that it's possible for log start offset retention to unblock further deletions as a result of time based retention. Note that this does require a case where the max timestamp for each segment increases, decreases and then increases again. Even so it would be nice to make retention idempotent by applying log start offset retention first, followed by size and time. This would also be potentially cheaper to perform as neither log start offset and size retention require the maxTimestamp for a segment to be loaded from disk after a broker restart.

Issue Links

• links to: GitHub Pull Request #10216