[KAFKA-7889] Kafka broker crashes on windows when log compaction is used

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.0, 1.1.1, 2.1.0
• Component/s: log
• Assignee: Unassigned
• Reporter: huzaifa kagazwala
• Created: 31/Jan/19 19:26
• Updated: 30/Jan/20 11:21

Description

I am running kafka on a Windows 10 machine tried the latest "kafka_2.12-2.1.0" build. I created a topic with log compaction enabled for testing. After publishing some data as soon as the first log rollover happens the server crashes with the below error.

 

command used to create the topic.

kafka-topics.bat -create -zookeeper localhost:2181 -topic compact_log__test_2  -replication-factor 1 -partitions 2 -config segment.bytes=100000 -config segment.ms=600000 -config min.cleanable.dirty.ratio=0.01 -config compression.type=snappy -config cleanup.policy=compact -config min.compaction.lag.ms=30000

 

Error Details:

 

[2019-01-31 11:12:09,763] INFO [ProducerStateManager partition=compact_log__test_2-0] Writing producer snapshot at offset 998 (kafka.log.ProducerStateManager)
[2019-01-31 11:12:09,770] INFO [Log partition=compact_log__test_2-0, dir=C:\kafka_logs] Rolled new log segment at offset 998 in 35 ms. (kafka.log.Log)
[2019-01-31 11:12:09,791] INFO [ProducerStateManager partition=compact_log__test_2-1] Writing producer snapshot at offset 1002 (kafka.log.ProducerStateManager)
[2019-01-31 11:12:09,796] INFO [Log partition=compact_log__test_2-1, dir=C:\kafka_logs] Rolled new log segment at offset 1002 in 17 ms. (kafka.log.Log)
[2019-01-31 11:12:34,711] ERROR Failed to clean up log for compact_log__test_2-0 in dir C:\kafka_logs due to IOException (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: C:\kafka_logs\compact_log_test_2-0\00000000000000000000.timeindex.cleaned -> C:\kafka_logs\compact_log_test_2-0\00000000000000000000.timeindex.swap: The process cannot access the file because it is being used by another process.

at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:809)
at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:205)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:490)
at kafka.log.Log.$anonfun$replaceSegments$4(Log.scala:1892)
at kafka.log.Log.$anonfun$replaceSegments$4$adapted(Log.scala:1892)
at scala.collection.immutable.List.foreach(List.scala:388)
at kafka.log.Log.replaceSegments(Log.scala:1892)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:583)
at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:515)
at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:514)
at scala.collection.immutable.List.foreach(List.scala:388)
at kafka.log.Cleaner.doClean(LogCleaner.scala:514)
at kafka.log.Cleaner.clean(LogCleaner.scala:492)
at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:353)
at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:319)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:300)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Suppressed: java.nio.file.FileSystemException: C:\kafka_logs\compact_log_test_2-0\00000000000000000000.timeindex.cleaned -> C:\kafka_logs\compact_log_test_2-0\00000000000000000000.timeindex.swap: The process cannot access the file because it is being used by another process.

at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:806)
... 16 more
[2019-01-31 11:12:34,716] INFO [ReplicaManager broker=0] Stopping serving replicas in dir C:\kafka_logs (kafka.server.ReplicaManager)
[2019-01-31 11:12:34,723] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(_consumer_offsets-22, __consumer_offsets-30, __consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9, __consumer_offsets-46, __consumer_offsets-25, compact_logtest_2-0, __consumer_offsets-35, __consumer_offsets-41, __consumer_offsets-33, __consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, __consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31, __consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3, __consumer_offsets-18, __consumer_offsets-37, __consumer_offsets-15, __consumer_offsets-24, __consumer_offsets-38, __consumer_offsets-17, __consumer_offsets-48, __consumer_offsets-19, __consumer_offsets-11, __consumer_offsets-13, __consumer_offsets-2, __consumer_offsets-43, __consumer_offsets-6, __consumer_offsets-14, compact_log_test_2-1, __consumer_offsets-20, __consumer_offsets-0, __consumer_offsets-44, __consumer_offsets-39, __consumer_offsets-12, __consumer_offsets-45, __consumer_offsets-1, __consumer_offsets-5, __consumer_offsets-26, __consumer_offsets-29, __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-32, __consumer_offsets-40) (kafka.server.ReplicaFetcherManager)
[2019-01-31 11:12:34,724] INFO [ReplicaAlterLogDirsManager on broker 0] Removed fetcher for partitions Set(_consumer_offsets-22, __consumer_offsets-30, __consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9, __consumer_offsets-46, __consumer_offsets-25, compact_logtest_2-0, __consumer_offsets-35, __consumer_offsets-41, __consumer_offsets-33, __consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, __consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31, __consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3, __consumer_offsets-18, __consumer_offsets-37, __consumer_offsets-15, __consumer_offsets-24, __consumer_offsets-38, __consumer_offsets-17, __consumer_offsets-48, __consumer_offsets-19, __consumer_offsets-11, __consumer_offsets-13, __consumer_offsets-2, __consumer_offsets-43, __consumer_offsets-6, __consumer_offsets-14, compact_log_test_2-1, __consumer_offsets-20, __consumer_offsets-0, __consumer_offsets-44, __consumer_offsets-39, __consumer_offsets-12, __consumer_offsets-45, __consumer_offsets-1, __consumer_offsets-5, __consumer_offsets-26, __consumer_offsets-29, __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-32, __consumer_offsets-40) (kafka.server.ReplicaAlterLogDirsManager)
[2019-01-31 11:12:34,751] INFO [ReplicaManager broker=0] Broker 0 stopped fetcher for partitions _consumer_offsets-22,consumer_offsets-30,consumer_offsets-8,consumer_offsets-21,consumer_offsets-4,consumer_offsets-27,consumer_offsets-7,consumer_offsets-9,consumer_offsets-46,consumer_offsets-25,compact_logtest_2-0,consumer_offsets-35,consumer_offsets-41,consumer_offsets-33,consumer_offsets-23,consumer_offsets-49,consumer_offsets-47,consumer_offsets-16,consumer_offsets-28,consumer_offsets-31,consumer_offsets-36,consumer_offsets-42,consumer_offsets-3,consumer_offsets-18,consumer_offsets-37,consumer_offsets-15,consumer_offsets-24,consumer_offsets-38,consumer_offsets-17,consumer_offsets-48,consumer_offsets-19,consumer_offsets-11,consumer_offsets-13,consumer_offsets-2,consumer_offsets-43,consumer_offsets-6,consumer_offsets-14,compact_logtest_2-1,consumer_offsets-20,consumer_offsets-0,consumer_offsets-44,consumer_offsets-39,consumer_offsets-12,consumer_offsets-45,consumer_offsets-1,consumer_offsets-5,consumer_offsets-26,consumer_offsets-29,consumer_offsets-34,consumer_offsets-10,consumer_offsets-32,_consumer_offsets-40 and stopped moving logs for partitions because they are in the failed log directory C:\kafka_logs. (kafka.server.ReplicaManager)
[2019-01-31 11:12:34,753] INFO Stopping serving logs in dir C:\kafka_logs (kafka.log.LogManager)
[2019-01-31 11:12:34,759] ERROR Shutdown broker because all log dirs in C:\kafka_logs have failed (kafka.log.LogManager)

Issue Links

• relates to: KAFKA-9458 Kafka crashed in windows environment