Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7061

Enhanced log compaction

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 2.5.0
    • None
    • core

    Description

      Enhance log compaction to support more than just offset comparison, so the insertion order isn't dictating which records to keep.

      Default behavior is kept as it was, with the enhanced approached having to be purposely activated.
      The enhanced compaction is done either via the record timestamp, by settings the new configuration as "timestamp" or via the record headers by setting this configuration to anything other than the default "offset" or the reserved "timestamp".

      See KIP-280 for more details.

      From Guozhang: We should emphasize on the WIKI that the newly introduced config yields to the existing "log.cleanup.policy", i.e. if the latter's value is `delete` not `compact`, then the previous config would be ignored.

      From Jun Rao: With the timestamp/header strategy, the behavior of the application may need to change. In particular, the application can't just blindly take the record with a larger offset and assuming that it's the value to keep. It needs to check the timestamp or the header now. So, it would be useful to at least document this. 

      Attachments

        Issue Links

          Activity

            People

              senthilm-ms Senthilnathan Muthusamy
              blaghed Luis Cabral
              Votes:
              5 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated: