Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-12177

Retention is not idempotent

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Kafka today applies retention in the following order:

      1. Time
      2. Size
      3. Log start offset

      Today it is possible for a segment with offsets less than the log start offset to contain data that is not deletable due to time retention. This means that it's possible for log start offset retention to unblock further deletions as a result of time based retention. Note that this does require a case where the max timestamp for each segment increases, decreases and then increases again. Even so it would be nice to make retention idempotent by applying log start offset retention first, followed by size and time. This would also be potentially cheaper to perform as neither log start offset and size retention require the maxTimestamp for a segment to be loaded from disk after a broker restart.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lucasbradstreet Lucas Bradstreet
                Reporter:
                lucasbradstreet Lucas Bradstreet
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: