Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6059

Kafka cant delete old log files on windows

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1, 0.10.2.0, 0.10.2.1, 0.11.0.0
    • Fix Version/s: None
    • Component/s: log
    • Labels:
    • Environment:
      OS:windows 2016
      kafka:10.0.2.1
      zookeeper:3.5.2

      Description

      I had a trouble for delete old log files.
      Now , I had disk space usage issues.

      I found exception from kafka's log and content as below:
      kafka.common.KafkaStorageException: Failed to change the log file suffix from to .deleted for log segment 0
      at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:340)
      at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:344)
      at kafka.log.Log.asyncDeleteSegment(Log.scala:981)
      at kafka.log.Log.deleteSegment(Log.scala:971)
      at kafka.log.Log.$anonfun$deleteOldSegments$1(Log.scala:673)
      at kafka.log.Log.$anonfun$deleteOldSegments$1$adapted(Log.scala:673)
      at scala.collection.immutable.List.foreach(List.scala:378)
      at kafka.log.Log.deleteOldSegments(Log.scala:673)
      at kafka.log.Log.deleteRetenionMsBreachedSegments(Log.scala:703)
      at kafka.log.Log.deleteOldSegments(Log.scala:697)
      at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:474)
      at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:472)
      at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
      at scala.collection.Iterator.foreach(Iterator.scala:929)
      at scala.collection.Iterator.foreach$(Iterator.scala:929)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1406)
      at scala.collection.IterableLike.foreach(IterableLike.scala:71)
      at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
      at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
      at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
      at kafka.log.LogManager.cleanupLogs(LogManager.scala:472)
      at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:200)
      at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
      at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.nio.file.FileSystemException: C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log -> C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

      at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
      at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
      at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
      at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
      at java.nio.file.Files.move(Files.java:1395)
      at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:711)
      at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:210)
      at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:342)
      ... 29 more
      Suppressed: java.nio.file.FileSystemException: C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log -> C:\kafka\kafka-logs\telegraf1-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

      at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
      at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
      at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
      at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
      at java.nio.file.Files.move(Files.java:1395)
      at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:708)
      ... 31 more

      does anyone can fix it?

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rico rico

              Dates

              • Created:
                Updated:

                Issue deployment