[2018-10-27 07:00:06,286] INFO Deleting segment 15026549356 from log __consumer_offsets-24. (kafka.log.Log) [2018-10-27 07:00:06,286] INFO Deleting index /data/kafka-logs/__consumer_offsets-24/00000000015026549356.index.deleted (kafka.log.OffsetIndex) ... [2018-10-27 07:00:06,295] INFO Deleting index /data/kafka-logs/__consumer_offsets-24/00000000015026549356.timeindex.deleted (kafka.log.TimeIndex) ... [2018-10-27 07:02:19,835] INFO Deleting segment 15026549356 from log __consumer_offsets-24. (kafka.log.Log) [2018-10-27 07:02:19,835] INFO Deleting index /data/kafka-logs/__consumer_offsets-24/00000000015026549356.index.deleted (kafka.log.OffsetIndex) [2018-10-27 07:02:19,835] INFO Deleting index /data/kafka-logs/__consumer_offsets-24/00000000015026549356.timeindex.deleted (kafka.log.TimeIndex) ... [2018-10-27 07:02:50,059] INFO Deleting segment 15026549356 from log __consumer_offsets-24. (kafka.log.Log) [2018-10-27 07:02:50,059] INFO Deleting index /data/kafka-logs/__consumer_offsets-24/00000000015026549356.index.deleted (kafka.log.OffsetIndex) [2018-10-27 07:02:50,059] INFO Deleting index /data/kafka-logs/__consumer_offsets-24/00000000015026549356.timeindex.deleted (kafka.log.TimeIndex) ... [2018-10-27 07:02:50,071] ERROR Error while deleting segments for __consumer_offsets-24 in dir /data/kafka-logs (kafka.server.LogDirFailureChannel) java.io.IOException: Delete of log 00000000015026549356.log.deleted failed. at kafka.log.LogSegment.delete(LogSegment.scala:496) at kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply$mcV$sp(Log.scala:1596) at kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596) at kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596) at kafka.log.Log.maybeHandleIOException(Log.scala:1669) at kafka.log.Log.kafka$log$Log$$deleteSeg$1(Log.scala:1595) at kafka.log.Log$$anonfun$kafka$log$Log$$asyncDeleteSegment$1.apply$mcV$sp(Log.scala:1599) at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2018-10-27 07:02:50,089] INFO [ReplicaManager broker=5] Stopping serving replicas in dir /data/kafka-logs (kafka.server.ReplicaManager) [2018-10-27 07:02:50,089] ERROR Uncaught exception in scheduled task 'delete-file' (kafka.utils.KafkaScheduler) org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for __consumer_offsets-24 in dir /data/kafka-logs Caused by: java.io.IOException: Delete of log 00000000015026549356.log.deleted failed. at kafka.log.LogSegment.delete(LogSegment.scala:496) at kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply$mcV$sp(Log.scala:1596) at kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596) at kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596) at kafka.log.Log.maybeHandleIOException(Log.scala:1669) at kafka.log.Log.kafka$log$Log$$deleteSeg$1(Log.scala:1595) at kafka.log.Log$$anonfun$kafka$log$Log$$asyncDeleteSegment$1.apply$mcV$sp(Log.scala:1599) at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ... [2018-10-27 07:02:50,124] ERROR Error while processing data for partition topic_redacted-0 (kafka.server.ReplicaFetcherThread) org.apache.kafka.common.errors.KafkaStorageException: Replica 5 is in an offline log directory for partition topic_redacted-0 ... [2018-10-27 07:02:50,767] WARN [ReplicaManager broker=5] While recording the replica LEO, the partition topic_redacted-65 hasn't been created. (kafka.server.ReplicaManager) ... [2018-10-27 07:02:51,548] INFO Stopping serving logs in dir /data/kafka-logs (kafka.log.LogManager) [2018-10-27 07:02:51,569] FATAL Shutdown broker because all log dirs in /data/kafka-logs have failed (kafka.log.LogManager)