[KAFKA-9970] Kafka 2.4.0 crashes under Windows when client is restarted

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0
• Component/s: core
• Assignee: Unassigned
• Reporter: SledgeHammer
• Created: 07/May/20 20:45
• Updated: 07/May/20 20:45

Description

Windows 10 x64 Pro

JDK 11.0.7

Zookeeper 3.6.0

Kafka 2.12-2.4.0

 

I have reproduced this scenario on multiple machines. I do dev on my Windows box, so have ZK and K running locally. On my work PC, I'll leave ZK & K running in command windows for "ever" at home I'll shutdown when I'm not doing dev.

In either case, Spring Boot client is continuously started and restarted. Intermittently Kafka will crash and corrupt the logs (I'm not able to capture the K crash exception since it closes), but upon restart, I get the exception below. NOTE: file is not in use since I can delete the logs directory and then restart. Client is both streams and classic queues.

 

[2020-05-07 13:38:27,782] ERROR Failed to clean up log for __consumer_offsets-20 in dir C:\PROGRA~1\kafka_2.12-2.4.0\logs due to IOException (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: C:\PROGRA~1\kafka_2.12-2.4.0\logs__consumer_offsets-20\00000000000000000000.timeindex.cleaned -> C:\PROGRA~1\kafka_2.12-2.4.0\logs__consumer_offsets-20\00000000000000000000.timeindex.swap: The process cannot access the file because it is being used by another process.

at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:395)
at java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:292)
at java.base/java.nio.file.Files.move(Files.java:1421)
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:795)
at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:209)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:497)
at kafka.log.Log.$anonfun$replaceSegments$4(Log.scala:2267)
at kafka.log.Log.$anonfun$replaceSegments$4$adapted(Log.scala:2267)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.log.Log.replaceSegments(Log.scala:2267)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:604)
at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:529)
at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:528)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.log.Cleaner.doClean(LogCleaner.scala:528)
at kafka.log.Cleaner.clean(LogCleaner.scala:502)
at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:371)
at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:344)
at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:324)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:313)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)