[KAFKA-3330] Truncate log cleaner offset checkpoint if the log is truncated

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.10.0.0
• Assignee: Dong Lin
• Reporter: Dong Lin
• Reviewer: Jun Rao
• Created: 03/Mar/16 22:17
• Updated: 23/Jul/18 23:39
• Resolved: 17/Mar/16 22:21

Description

Were getting a number of failures of the log compaction thread with the
following error:
2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
__consumer_offsets-93.
2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
__consumer_offsets-93...
2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
__consumer_offsets-93 for 2 segments in offset range [11951210572,
11952632314).
2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
due to
java.lang.IllegalArgumentException: requirement failed: Last clean offset
is 11951210572 but segment base offset is 11950300163 for log
__consumer_offsets-93.
at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.log.Cleaner.clean(LogCleaner.scala:306)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
~[kafka_2.10-0.8.2.56.jar:?]
2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped

We found that this may be caused in the following scenario:

• we have three log segments with offset range [100, 200), [200, 300), and [300, 400) respectively. 300 is the base offset of the active log segment. Log cleaner offset checkpoint is also 300.

• After log is truncated to offset 220, the log segments become [100, 200), [200, 220). The Log cleaner offset checkpoint is still 300.

• After new messages are appended to the log, the log segments become [100, 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.

• Log cleaner cleans the log starting at offset 300. The require(offset == start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not the base offset of any segments.

To fix the problem, when the log is truncated to an offset smaller than cleaner offset checkpoint, we should reset cleaner offset checkpoint to the base offset of the active segment if this value is larger than the checkpointed offset.

Issue Links

• mentioned in: Page Loading...