Flume
  1. Flume
  2. FLUME-2119

duplicate files cause flume to enter irrecoverable state

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      If a spoolingdir receives FileA, after it is picked up by Flume and renamed to FileA.COMPLETED placing another file of the same original name (FileA) will cause Flume to log an IllegalStateException indefinitely. This is likely due to Flume attempting to rename the second FileA to FileA.COMPLETED, but finding that the file already exists.

      When Flume has entered this state, it can only be recovered by removing the .COMPLETED file from the directory and restarting the agent.

      Log message looks like this:

      02 Jul 2013 21:32:09,371 ERROR [pool-4-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:164) - Uncaught exception in Runnable
      java.lang.IllegalStateException: Serializer has been closed
      at org.apache.flume.serialization.LineDeserializer.ensureOpen(LineDeserializer.java:124)
      at org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:88)
      at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221)
      at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

        Issue Links

          Activity

            People

            • Assignee:
              Phil Scala
              Reporter:
              Jonathan Cooper-Ellis
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development