Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2012

Broker should automatically handle corrupt index files

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.1.1
    • Fix Version/s: 0.9.0.0
    • Component/s: None
    • Labels:
      None

      Description

      We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading).

      The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it.

      2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown
      java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/00000000000000000000.index) has non-zero size but the last offset is -2121629628 and the base offset is 0
      at scala.Predef$.require(Predef.scala:233)
      at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
      at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185)
      at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
      at scala.collection.Iterator$class.foreach(Iterator.scala:727)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
      at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
      at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
      at kafka.log.Log.loadSegments(Log.scala:184)
      at kafka.log.Log.<init>(Log.scala:82)
      at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141)
      at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

        Attachments

        1. KAFKA-2012.patch
          3 kB
          Manikumar
        2. KAFKA-2012_2015-06-19_18:55:11.patch
          4 kB
          Manikumar
        3. KAFKA-2012_2015-06-19_21:09:22.patch
          4 kB
          Manikumar

          Issue Links

            Activity

              People

              • Assignee:
                omkreddy Manikumar
                Reporter:
                toddpalino Todd Palino
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: