Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3330

Truncate log cleaner offset checkpoint if the log is truncated

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Were getting a number of failures of the log compaction thread with the
      following error:
      2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
      __consumer_offsets-93.
      2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
      __consumer_offsets-93...
      2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
      __consumer_offsets-93 for 2 segments in offset range [11951210572,
      11952632314).
      2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
      due to
      java.lang.IllegalArgumentException: requirement failed: Last clean offset
      is 11951210572 but segment base offset is 11950300163 for log
      __consumer_offsets-93.
      at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
      at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
      ~[kafka_2.10-0.8.2.56.jar:?]
      at kafka.log.Cleaner.clean(LogCleaner.scala:306)
      ~[kafka_2.10-0.8.2.56.jar:?]
      at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
      ~[kafka_2.10-0.8.2.56.jar:?]
      at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
      ~[kafka_2.10-0.8.2.56.jar:?]
      at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
      ~[kafka_2.10-0.8.2.56.jar:?]
      2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped

      We found that this may be caused in the following scenario:

      • we have three log segments with offset range [100, 200), [200, 300), and [300, 400) respectively. 300 is the base offset of the active log segment. Log cleaner offset checkpoint is also 300.
      • After log is truncated to offset 220, the log segments become [100, 200), [200, 220). The Log cleaner offset checkpoint is still 300.
      • After new messages are appended to the log, the log segments become [100, 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
      • Log cleaner cleans the log starting at offset 300. The require(offset == start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not the base offset of any segments.

      To fix the problem, when the log is truncated to an offset smaller than cleaner offset checkpoint, we should reset cleaner offset checkpoint to the base offset of the active segment if this value is larger than the checkpointed offset.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lindong Dong Lin
                Reporter:
                lindong Dong Lin
                Reviewer:
                Jun Rao
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: