Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3330

Truncate log cleaner offset checkpoint if the log is truncated



    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s:
    • Component/s: None
    • Labels:


      Were getting a number of failures of the log compaction thread with the
      following error:
      2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
      2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
      2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
      __consumer_offsets-93 for 2 segments in offset range [11951210572,
      2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
      due to
      java.lang.IllegalArgumentException: requirement failed: Last clean offset
      is 11951210572 but segment base offset is 11950300163 for log
      at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
      at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
      at kafka.log.Cleaner.clean(LogCleaner.scala:306)
      at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
      at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
      at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
      2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped

      We found that this may be caused in the following scenario:

      • we have three log segments with offset range [100, 200), [200, 300), and [300, 400) respectively. 300 is the base offset of the active log segment. Log cleaner offset checkpoint is also 300.
      • After log is truncated to offset 220, the log segments become [100, 200), [200, 220). The Log cleaner offset checkpoint is still 300.
      • After new messages are appended to the log, the log segments become [100, 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
      • Log cleaner cleans the log starting at offset 300. The require(offset == start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not the base offset of any segments.

      To fix the problem, when the log is truncated to an offset smaller than cleaner offset checkpoint, we should reset cleaner offset checkpoint to the base offset of the active segment if this value is larger than the checkpointed offset.


          Issue Links



              • Assignee:
                lindong Dong Lin
                lindong Dong Lin
                Jun Rao
              • Votes:
                0 Vote for this issue
                5 Start watching this issue


                • Created: