Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2119

duplicate files cause flume to enter irrecoverable state

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.4.0
    • None
    • Sinks+Sources
    • None

    Description

      If a spoolingdir receives FileA, after it is picked up by Flume and renamed to FileA.COMPLETED placing another file of the same original name (FileA) will cause Flume to log an IllegalStateException indefinitely. This is likely due to Flume attempting to rename the second FileA to FileA.COMPLETED, but finding that the file already exists.

      When Flume has entered this state, it can only be recovered by removing the .COMPLETED file from the directory and restarting the agent.

      Log message looks like this:

      02 Jul 2013 21:32:09,371 ERROR [pool-4-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:164) - Uncaught exception in Runnable
      java.lang.IllegalStateException: Serializer has been closed
      at org.apache.flume.serialization.LineDeserializer.ensureOpen(LineDeserializer.java:124)
      at org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:88)
      at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221)
      at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

      Attachments

        1. FLUME-2119-0.patch
          24 kB
          Phil Scala

        Issue Links

          Activity

            People

              scaph01 Phil Scala
              jcoopere Jonathan Cooper-Ellis
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: