[KAFKA-7951] Log Cleaner thread stop with "Varint is too long" error

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 0.11.0.2, 0.11.0.3
• Component/s: log
• Environment: Amazon Linux AMI 2017.09 ; confluent-platform-oss-2.11-3.3.2 ; OpenJDK Runtime Environment (build 1.8.0_191-b12) ; EC2 instance type i3.xlarge
• Assignee: Unassigned
• Reporter: __
• Created: 19/Feb/19 15:33
• Updated: 21/Feb/19 14:48

Description

I have one broker log cleaner thread dying because of a failing type conversion. My cluster is running on 0.11.03-cp1 (Confluent Platform). Here's the relevant logs in log-cleaner.log

[2019-02-19 15:08:23,387] priority=INFO message="[kafka-log-cleaner-thread-0]:
        Log cleaner thread 0 cleaned log __consumer_offsets-36 (dirty section = [931148954, 931148954])
        13,051.5 MB of log processed in 159.0 seconds (82.1 MB/sec).
        Indexed 13,051.5 MB in 81.2 seconds (160.8 Mb/sec, 51.1% of total time)
        Buffer utilization: 0.0%
        Cleaned 13,051.5 MB in 77.8 seconds (167.7 Mb/sec, 48.9% of total time)
        Start size: 13,051.5 MB (166,676,749 messages)
        End size: 0.0 MB (159 messages)
        100.0% size reduction (100.0% fewer messages)
" category=kafka.log.LogCleaner
[2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Beginning cleaning of log __consumer_offsets-37." category=kafka.log.LogCleaner
[2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Building offset map for __consumer_offsets-37..." category=kafka.log.LogCleaner
[2019-02-19 15:08:23,410] priority=INFO message="Cleaner 0: Building offset map for log __consumer_offsets-37 for 87 segments in offset range [211566448, 308245257)." category=kafka.log.LogCleaner
[2019-02-19 15:08:23,945] priority=ERROR message="[kafka-log-cleaner-thread-0]: Error due to" category=kafka.log.LogCleaner
java.lang.IllegalArgumentException: Varint is too long, the most significant bit in the 5th byte is set, converted value: 445d418c
        at org.apache.kafka.common.utils.ByteUtils.illegalVarintException(ByteUtils.java:326)
        at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:148)
        at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:305)
        at org.apache.kafka.common.record.DefaultRecordBatch$2.readNext(DefaultRecordBatch.java:299)
        at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563)
        at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532)
        at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:752)
        at kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:741)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:741)
        at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:707)
        at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:704)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:704)
        at kafka.log.Cleaner.doClean(LogCleaner.scala:373)
        at kafka.log.Cleaner.clean(LogCleaner.scala:361)
        at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:256)
        at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:236)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
[2019-02-19 15:08:23,964] priority=INFO message="[kafka-log-cleaner-thread-0]: Stopped" category=kafka.log.LogCleaner

This is the first time we are seeing this error and coincidentally we have changed our JVM from Oracle to OpenJDK, both 1.8 Java.

 

Please note that I can't upgrade to newer version of Kafka yet.

Thanks!