Kafka
  1. Kafka
  2. KAFKA-70

Introduce retention setting that depends on space

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: None
    • Labels:

      Description

      Currently there is this setting:

      log.retention.hours

      introduce:

      log.retention.size

      Semantic would be, either size is reached or time is reached, oldest messages will be deleted.

      This does not break back-ward compatibility and would make the system robust under scenarios where message size is not deterministic over time.

      1. KAFKA-70_v2.patch
        7 kB
        Prashanth Menon
      2. KAFKA-70.patch
        7 kB
        Prashanth Menon

        Activity

        Jun Rao made changes -
        Fix Version/s 0.7 [ 12317243 ]
        Fix Version/s 0.8 [ 12317244 ]
        Reporter Jun Rao [ junrao ]
        Affects Version/s 0.7 [ 12317243 ]
        Affects Version/s 0.8 [ 12317244 ]
        Jay Kreps made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.8 [ 12317244 ]
        Resolution Unresolved [ 9 ] Fixed [ 1 ]
        Prashanth Menon made changes -
        Comment [ Updated patch. ]
        Prashanth Menon made changes -
        Attachment KAFKA-70_v2.patch [ 12494165 ]
        Prashanth Menon made changes -
        Attachment KAFKA-70.patch.v2 [ 12494164 ]
        Prashanth Menon made changes -
        Attachment KAFKA-70.patch.v2 [ 12494164 ]
        Prashanth Menon made changes -
        Comment [ Patch against SVN trunk to enable this feature. ]
        Prashanth Menon made changes -
        Attachment KAFKA-70.patch [ 12493706 ]
        Prashanth Menon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.8 [ 12317244 ]
        Labels log roll
        Prashanth Menon made changes -
        Comment [ Hi there, I'm currently looking into this issue (and the larger project as a whole) and have a few questions:

        - In the general case, either the time limit is reached and we trim, or the size limit is reached and we trim. What happens when we cleanup expired segments but still go over the size restraint? Do we continue by trimming the size or wait for the next cycle to check? I'd presume so but thought to make sure.
        - In the config, since this would be a new optional field, what would be a reasonable default for the retention size? I'm not sure what types of log file sizes the system deals with so any suggestions would be good.

        Thanks everyone!

        - Prashanth ]
        Alan Cabrera made changes -
        Field Original Value New Value
        Workflow jira [ 12621023 ] no-reopen-closed, patch-avail [ 12626224 ]
        Anonymous created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Jun Rao
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development