[KAFKA-13596] ERROR Error while deleting segments for {topic} : java.nio.file.NoSuchFileException

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.1
• Component/s: log
• Environment: kafka 2.0.1 on Centos 7.5 ; jvm: Java(TM) SE Runtime Environment (build 1.8.0_131)
• Assignee: Unassigned
• Reporter: shiyc
• Created: 14/Jan/22 03:12
• Updated: 19/Jan/22 03:25

Description

Our kafka server (version 2.0.1) failed because of a NoSuchFileException while it is trying to delete log files. The exceptions in the kafka log are :

ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for zabbix_topic_131-0 in dir /home/netvip/data/kafka
Caused by: java.nio.file.NoSuchFileException: /home/netvip/data/kafka/zabbix_topic_131-0/00000000000745417527.log
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:786)
    at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:211)
    at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:488)
    at kafka.log.Log.asyncDeleteSegment(Log.scala:1753)
    at kafka.log.Log.deleteSegment(Log.scala:1738)
    at kafka.log.Log.$anonfun$deleteSegments$3(Log.scala:1309)
    at kafka.log.Log.$anonfun$deleteSegments$3$adapted(Log.scala:1309)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1309)
    at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
    at kafka.log.Log.maybeHandleIOException(Log.scala:1842)
    at kafka.log.Log.deleteSegments(Log.scala:1300)
    at kafka.log.Log.deleteOldSegments(Log.scala:1295)
    at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1368)
    at kafka.log.Log.deleteOldSegments(Log.scala:1361)
    at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:874)
    at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:872)
    at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
    at scala.collection.immutable.List.foreach(List.scala:389)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
    at kafka.log.LogManager.cleanupLogs(LogManager.scala:872)
    at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:395)
    at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
    at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: java.nio.file.NoSuchFileException: /home/netvip/data/kafka/zabbix_topic_131-0/00000000000745417527.log -> /home/netvip/data/kafka/zabbix_topic_131-0/00000000000745417527.log.deleted
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
        at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
        at java.nio.file.Files.move(Files.java:1395)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:783)
        ... 32 more 

The file '00000000000745417527.log' does not exist on disk.