Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7951

Log Cleaner thread stop with "Varint is too long" error

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 0.11.0.2, 0.11.0.3
    • Fix Version/s: None
    • Component/s: log
    • Labels:
      None
    • Environment:
      Amazon Linux AMI 2017.09
      confluent-platform-oss-2.11-3.3.2
      OpenJDK Runtime Environment (build 1.8.0_191-b12)
      EC2 instance type i3.xlarge

      Description

      I have one broker log cleaner thread dying because of a failing type conversion. My cluster is running on 0.11.03-cp1 (Confluent Platform). Here's the relevant logs in log-cleaner.log

      [2019-02-19 15:08:23,387] priority=INFO message="[kafka-log-cleaner-thread-0]:
              Log cleaner thread 0 cleaned log __consumer_offsets-36 (dirty section = [931148954, 931148954])
              13,051.5 MB of log processed in 159.0 seconds (82.1 MB/sec).
              Indexed 13,051.5 MB in 81.2 seconds (160.8 Mb/sec, 51.1% of total time)
              Buffer utilization: 0.0%
              Cleaned 13,051.5 MB in 77.8 seconds (167.7 Mb/sec, 48.9% of total time)
              Start size: 13,051.5 MB (166,676,749 messages)
              End size: 0.0 MB (159 messages)
              100.0% size reduction (100.0% fewer messages)
      " category=kafka.log.LogCleaner
      [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Beginning cleaning of log __consumer_offsets-37." category=kafka.log.LogCleaner
      [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Building offset map for __consumer_offsets-37..." category=kafka.log.LogCleaner
      [2019-02-19 15:08:23,410] priority=INFO message="Cleaner 0: Building offset map for log __consumer_offsets-37 for 87 segments in offset range [211566448, 308245257)." category=kafka.log.LogCleaner
      [2019-02-19 15:08:23,945] priority=ERROR message="[kafka-log-cleaner-thread-0]: Error due to" category=kafka.log.LogCleaner
      java.lang.IllegalArgumentException: Varint is too long, the most significant bit in the 5th byte is set, converted value: 445d418c
              at org.apache.kafka.common.utils.ByteUtils.illegalVarintException(ByteUtils.java:326)
              at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:148)
              at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:305)
              at org.apache.kafka.common.record.DefaultRecordBatch$2.readNext(DefaultRecordBatch.java:299)
              at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563)
              at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532)
              at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
              at scala.collection.Iterator$class.foreach(Iterator.scala:891)
              at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
              at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
              at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
              at kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:752)
              at kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:741)
              at scala.collection.Iterator$class.foreach(Iterator.scala:891)
              at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
              at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
              at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
              at kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:741)
              at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:707)
              at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:704)
              at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
              at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
              at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
              at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
              at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:704)
              at kafka.log.Cleaner.doClean(LogCleaner.scala:373)
              at kafka.log.Cleaner.clean(LogCleaner.scala:361)
              at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:256)
              at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:236)
              at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
      [2019-02-19 15:08:23,964] priority=INFO message="[kafka-log-cleaner-thread-0]: Stopped" category=kafka.log.LogCleaner

      This is the first time we are seeing this error and coincidentally we have changed our JVM from Oracle to OpenJDK, both 1.8 Java.

       

      Please note that I can't upgrade to newer version of Kafka yet.

      Thanks!

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ninth.nails Ninth Nails
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: