[KAFKA-4790] Kafka cannot recover after a disk full

• Type: Bug
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0.1, 0.10.1.1, 2.1.0
• Fix Version/s: 0.11.0.0
• Labels: reliability
• Assignee: Unassigned
• Reporter: Pengwei
• Created: 23/Feb/17 12:58
• Updated: 19/Jul/19 14:35

Description

[2017-02-23 18:43:57,736] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2017-02-23 18:43:57,887] INFO Loading logs. (kafka.log.LogManager)
[2017-02-23 18:43:57,935] INFO Recovering unflushed segment 0 in log test1-0. (kafka.log.Log)
[2017-02-23 18:43:59,297] ERROR There was an error in one of the threads during logs loading: java.lang.IllegalArgumentException: requirement failed: Attempt to append to a full index (size = 128000). (kafka.log.LogManager)
[2017-02-23 18:43:59,299] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: requirement failed: Attempt to append to a full index (size = 128000).
at scala.Predef$.require(Predef.scala:219)
at kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:200)
at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:199)
at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:199)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
at kafka.log.OffsetIndex.append(OffsetIndex.scala:199)
at kafka.log.LogSegment.recover(LogSegment.scala:191)
at kafka.log.Log.recoverLog(Log.scala:259)
at kafka.log.Log.loadSegments(Log.scala:234)
at kafka.log.Log.<init>(Log.scala:92)
at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$4$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:201)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Issue Links

• relates to: KAFKA-4816 Message format changes for idempotent/transactional producer