Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1791

Corrupt index after safe shutdown and restart

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Affects Version/s: 0.8.1
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None
    • Environment:
      Debian6 with Sun-Java6

      Description

      We have 3 kafka brokers - all VMs. One of the broker was stopped for around 30 minutes to fix a problem with the bare metal. Upon restart, after some time, the broker went out of file-descriptors (FDs) and started throwing errors. Upon restart, it started throwing this corrupted index exceptions. I followed the other JIRAs related to corrupted indices but the solutions mentioned there (deleting the index and restart) didn't work - the index gets created again. The other JIRAs solution of deleting those indexes which got wrongly compacted (> 10MB) didn't work either.

      What is the error? How can I fix this and bring back the broker? Thanks.

      INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Found clean shutdown file. Skipping recovery for all logs in data directory '/var/lib/fk-3p-kafka/logs'
      INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Loading log 'kf.production.b2b.return_order.status-25'
      FATAL [2014-11-21 02:57:17,533] [main][] kafka.server.KafkaServerStartable - Fatal error during KafkaServerStable startup. Prepare to shutdown
      java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/var/lib/fk-3p-kafka/logs/kf.production.b2b.return_order.status-25/00000000000000000233.index) has non-zero size but the last offset is 233 and the base offset is 233
      at scala.Predef$.require(Predef.scala:145)
      at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
      at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159)
      at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158)
      at scala.collection.Iterator$class.foreach(Iterator.scala:631)
      at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474)
      at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
      at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495)
      at kafka.log.Log.loadSegments(Log.scala:158)
      at kafka.log.Log.<init>(Log.scala:64)
      at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
      at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
      at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
      at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
      at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
      at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
      at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
      at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
      at kafka.log.LogManager.loadLogs(LogManager.scala:105)
      at kafka.log.LogManager.<init>(LogManager.scala:57)
      at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
      at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
      at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
      at kafka.Kafka$.main(Kafka.scala:46)
      at kafka.Kafka.main(Kafka.scala)
      INFO [2014-11-21 02:57:17,534] [main][] kafka.server.KafkaServer - [Kafka Server 2], shutting down
      INFO [2014-11-21 02:57:17,538] [main][] kafka.server.KafkaServer - [Kafka Server 2], shut down completed
      INFO [2014-11-21 02:57:17,539] [Thread-2][] kafka.server.KafkaServer - [Kafka Server 2], shutting down

        Attachments

        1. 00000000000000000233.log
          13 kB
          Vamsi Subhash Achanta
        2. 00000000000000000233.index
          10.00 MB
          Vamsi Subhash Achanta

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vamsi360 Vamsi Subhash Achanta
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: