Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-555

Add a key-based log retention strategy

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      Currently we have two log retention strategies: one based on time and one based on log size. These work well for "event" type data--i.e. data that consists only of appends. However if the events model changes to an underlying keyed data set, a more convenient retention strategy would delete keys that had been overwritten rather than retaining whole segments.

      The proposed implementation would be a background process that scanned log segments and recopied only keys that hadn't been overwritten. Some more details are in this wiki:
      https://cwiki.apache.org/confluence/display/KAFKA/Keyed+Messages+Proposal

      Attachments

        Issue Links

          duplicates

          New Feature - A new feature of the product, which has yet to be developed. KAFKA-631 Implement log compaction

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              jkreps Jay Kreps
              jkreps Jay Kreps
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: