Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4576

Log segments close to max size break on fetch

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.10.1.1
    • Fix Version/s: 0.10.2.0
    • Component/s: log
    • Labels:

      Description

      We are running Kafka 0.10.1.1~rc1 (it's the same as 0.10.1.1).

      Max segment size is set to 2147483647 globally, that's 1 byte less than max signed int32.

      Every now and then we see failures like this:

      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: ERROR [Replica Manager on Broker 1006]: Error processing fetch operation on partition [mytopic,11], offset 483579108587 (kafka.server.ReplicaManager)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: java.lang.IllegalStateException: Failed to read complete buffer for targetOffset 483686627237 startPosition 2145701130 in /disk/data0/kafka-logs/mytopic-11/00000000483571890786.log
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.log.FileMessageSet.searchForOffsetWithSize(FileMessageSet.scala:145)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.log.LogSegment.translateOffset(LogSegment.scala:128)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.log.LogSegment.read(LogSegment.scala:180)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.log.Log.read(Log.scala:563)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.server.ReplicaManager.kafka$server$ReplicaManager$$read$1(ReplicaManager.scala:567)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:606)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:605)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at scala.collection.Iterator$class.foreach(Iterator.scala:893)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.server.ReplicaManager.readFromLocalLog(ReplicaManager.scala:605)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:469)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:534)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
      Dec 25 18:20:47 myhost kafka-run-class.sh[2054]:         at java.lang.Thread.run(Thread.java:745)
      
      ...
      -rw-r--r-- 1 kafka kafka          0 Dec 25 15:15 00000000483557418204.timeindex
      -rw-r--r-- 1 kafka kafka       9496 Dec 25 15:26 00000000483564654488.index
      -rw-r--r-- 1 kafka kafka 2145763964 Dec 25 15:26 00000000483564654488.log
      -rw-r--r-- 1 kafka kafka          0 Dec 25 15:26 00000000483564654488.timeindex
      -rw-r--r-- 1 kafka kafka       9576 Dec 25 15:37 00000000483571890786.index
      -rw-r--r-- 1 kafka kafka 2147483644 Dec 25 15:37 00000000483571890786.log
      -rw-r--r-- 1 kafka kafka          0 Dec 25 15:37 00000000483571890786.timeindex
      -rw-r--r-- 1 kafka kafka       9568 Dec 25 15:48 00000000483579135712.index
      -rw-r--r-- 1 kafka kafka 2146791360 Dec 25 15:48 00000000483579135712.log
      -rw-r--r-- 1 kafka kafka          0 Dec 25 15:48 00000000483579135712.timeindex
      -rw-r--r-- 1 kafka kafka       9408 Dec 25 15:59 00000000483586374164.index
      ...
      

      Here 00000000483571890786.log is just 3 bytes below the max size.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                huxi_2b huxihx
                Reporter:
                bobrik Ivan Babrou
                Reviewer:
                Ismael Juma
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: