Details
-
Bug
-
Status: Open
-
Blocker
-
Resolution: Unresolved
-
0.11.0.2, 0.11.0.3
-
None
-
None
-
Amazon Linux AMI 2017.09
confluent-platform-oss-2.11-3.3.2
OpenJDK Runtime Environment (build 1.8.0_191-b12)
EC2 instance type i3.xlarge
Description
I have one broker log cleaner thread dying because of a failing type conversion. My cluster is running on 0.11.03-cp1 (Confluent Platform). Here's the relevant logs in log-cleaner.log
[2019-02-19 15:08:23,387] priority=INFO message="[kafka-log-cleaner-thread-0]: Log cleaner thread 0 cleaned log __consumer_offsets-36 (dirty section = [931148954, 931148954]) 13,051.5 MB of log processed in 159.0 seconds (82.1 MB/sec). Indexed 13,051.5 MB in 81.2 seconds (160.8 Mb/sec, 51.1% of total time) Buffer utilization: 0.0% Cleaned 13,051.5 MB in 77.8 seconds (167.7 Mb/sec, 48.9% of total time) Start size: 13,051.5 MB (166,676,749 messages) End size: 0.0 MB (159 messages) 100.0% size reduction (100.0% fewer messages) " category=kafka.log.LogCleaner [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Beginning cleaning of log __consumer_offsets-37." category=kafka.log.LogCleaner [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Building offset map for __consumer_offsets-37..." category=kafka.log.LogCleaner [2019-02-19 15:08:23,410] priority=INFO message="Cleaner 0: Building offset map for log __consumer_offsets-37 for 87 segments in offset range [211566448, 308245257)." category=kafka.log.LogCleaner [2019-02-19 15:08:23,945] priority=ERROR message="[kafka-log-cleaner-thread-0]: Error due to" category=kafka.log.LogCleaner java.lang.IllegalArgumentException: Varint is too long, the most significant bit in the 5th byte is set, converted value: 445d418c at org.apache.kafka.common.utils.ByteUtils.illegalVarintException(ByteUtils.java:326) at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:148) at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:305) at org.apache.kafka.common.record.DefaultRecordBatch$2.readNext(DefaultRecordBatch.java:299) at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563) at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532) at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:752) at kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:741) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:741) at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:707) at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:704) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:704) at kafka.log.Cleaner.doClean(LogCleaner.scala:373) at kafka.log.Cleaner.clean(LogCleaner.scala:361) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:256) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:236) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64) [2019-02-19 15:08:23,964] priority=INFO message="[kafka-log-cleaner-thread-0]: Stopped" category=kafka.log.LogCleaner
This is the first time we are seeing this error and coincidentally we have changed our JVM from Oracle to OpenJDK, both 1.8 Java.
Please note that I can't upgrade to newer version of Kafka yet.
Thanks!