Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2556

Corrupted index on log.dir updates

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • 0.8.2.1
    • None
    • core
    • None

    Description

      Partition is corrupted when user updates server configuration.
      Topic would be corrupted if two or more `log.dir` directories contains segments for the same topic after configuration changes.
      Steps to reproduce:

      • Start Kafka service with several `log.dir` directories
      • Stop Kafka service
      • Update configuration. Remove one of `log.dir` directories
      • Start Kafka service. Kafka creates another directory for the partition in another directory
      • Stop Kafka service
      • Update configuration. Restore the folder we removed on the third step
      • Start Kafka process

      Result:

      [2015-09-18 10:28:48,764] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,801] INFO Property auto.create.topics.enable is overridden to false (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,801] INFO Property auto.leader.rebalance.enable is overridden to false (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,801] INFO Property broker.id is overridden to 6 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,802] INFO Property default.replication.factor is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,802] INFO Property host.name is overridden to inferno07.chi.net (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,803] WARN Property kafka.metrics.graphite.group is not valid (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,803] WARN Property kafka.metrics.graphite.host is not valid (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,803] WARN Property kafka.metrics.graphite.port is not valid (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,803] WARN Property kafka.metrics.graphite.regex is not valid (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,803] WARN Property kafka.metrics.polling.interval.secs is not valid (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,804] WARN Property kafka.metrics.reporters is not valid (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,804] WARN Property log.cleanup.interval.mins is not valid (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,804] INFO Property log.dirs is overridden to /mnt/21e57274-0ba1-41e0-9ea4-befef32ec62c/data/kafka,/mnt/f81fc52a-7784-47c3-8b6d-be95cc935697/data/kafka,/mnt/75928c58-289d-4954-8640-0b82a990b3bc/data/kafka,/mnt/6bcb0c8e-da33-43e6-b878-703600f622d1/data/kafka,/mnt/afd7361c-dc16-4b78-8608-adda119559d0/data/kafka,/mnt/96a93811-6d91-41aa-ab1a-3804f43a5b05/data/kafka,/mnt/c085ee04-65ff-493d-b236-fa3dd462595c/data/kafka,/mnt/fbec6169-ca70-4f07-a7e9-af85424a97c5/data/kafka,/mnt/363d4473-531f-446c-b483-49593587f965/data/kafka,/mnt/d47fd346-1b77-41a8-a0a8-1c517c394868/data/kafka,/mnt/2c97da3c-4d5d-49bf-ae5a-a46c2839b18a/data/kafka,/mnt/fc6fd370-a213-4b02-9a7c-07f7a5ad505f/data/kafka (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,804] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,804] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,805] INFO Property message.max.bytes is overridden to 41943040 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,805] INFO Property num.io.threads is overridden to 45 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,805] INFO Property num.network.threads is overridden to 150 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,805] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,806] INFO Property num.replica.fetchers is overridden to 10 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,806] INFO Property port is overridden to 9093 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,806] INFO Property replica.fetch.max.bytes is overridden to 41943040 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,806] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,806] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,807] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,808] INFO Property unclean.leader.election.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,808] INFO Property zookeeper.connect is overridden to inferno08.chi.net:32181,inferno09.chi.net:32181,inferno10.chi.net:32181,inferno11.chi.net:32181,inferno12.chi.net:32181, (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,808] INFO Property zookeeper.connection.timeout.ms is overridden to 10000 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,808] INFO Property zookeeper.session.timeout.ms is overridden to 10000 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,809] INFO Property zookeeper.sync.time.ms is overridden to 4000 (kafka.utils.VerifiableProperties)
[2015-09-18 10:28:48,874] INFO [Kafka Server 6], starting (kafka.server.KafkaServer)
[2015-09-18 10:28:48,877] INFO [Kafka Server 6], Connecting to zookeeper on inferno08.chi.net:32181,inferno09.chi.net:32181,inferno10.chi.net:32181,inferno11.chi.net:32181,inferno12.chi.net:32181, (kafka.server.KafkaServer)
[2015-09-18 10:28:49,295] INFO Loading logs. (kafka.log.LogManager)
[2015-09-18 10:28:49,375] INFO Recovering unflushed segment 1752968 in log pony-topic-73. (kafka.log.Log)
[2015-09-18 10:28:49,375] INFO Recovering unflushed segment 0 in log pony-topic-1. (kafka.log.Log)
[2015-09-18 10:28:49,375] INFO Recovering unflushed segment 1753474 in log pony-topic-13. (kafka.log.Log)
[2015-09-18 10:28:49,376] INFO Recovering unflushed segment 1753277 in log pony-topic-61. (kafka.log.Log)
[2015-09-18 10:28:49,376] INFO Recovering unflushed segment 1752741 in log pony-topic-85. (kafka.log.Log)
[2015-09-18 10:28:49,389] INFO Completed load of log pony-topic-1 with log end offset 0 (kafka.log.Log)
[2015-09-18 10:28:49,403] INFO Recovering unflushed segment 0 in log production-dropped-dtevents-topic-1. (kafka.log.Log)
[2015-09-18 10:28:50,950] INFO Completed load of log pony-topic-73 with log end offset 2017625 (kafka.log.Log)
[2015-09-18 10:28:50,967] INFO Completed load of log pony-topic-61 with log end offset 2020452 (kafka.log.Log)
[2015-09-18 10:28:50,989] INFO Completed load of log pony-topic-85 with log end offset 2019928 (kafka.log.Log)
[2015-09-18 10:28:50,996] INFO Completed load of log pony-topic-13 with log end offset 2018860 (kafka.log.Log)
[2015-09-18 10:28:53,222] INFO Completed load of log production-dropped-dtevents-topic-1 with log end offset 893236 (kafka.log.Log)
[2015-09-18 10:28:53,227] ERROR There was an error in one of the threads during logs loading: java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/mnt/21e57274-0ba1-41e0-9ea4-befef32ec62c/data/kafka/pony-topic-1/00000000000001753742.index) has non-zero size but the last offset is 1753742 and the base offset is 1753742 (kafka.log.LogManager)
[2015-09-18 10:28:53,230] FATAL [Kafka Server 6], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/mnt/21e57274-0ba1-41e0-9ea4-befef32ec62c/data/kafka/pony-topic-1/00000000000001753742.index) has non-zero size but the last offset is 1753742 and the base offset is 1753742
        at scala.Predef$.require(Predef.scala:219)
        at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
        at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184)
        at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:183)
        at scala.collection.Iterator$class.foreach(Iterator.scala:750)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1202)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at kafka.log.Log.loadSegments(Log.scala:183)
        at kafka.log.Log.<init>(Log.scala:67)
        at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
        at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2015-09-18 10:28:53,234] INFO [Kafka Server 6], shutting down (kafka.server.KafkaServer)
[2015-09-18 10:28:53,240] INFO [Kafka Server 6], shut down completed (kafka.server.KafkaServer)

      Attachments

        Activity

          People

            Unassigned Unassigned
            WonderBeat Denis Golovachev
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: