Flume
  1. Flume
  2. FLUME-2119

duplicate files cause flume to enter irrecoverable state

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.4.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      If a spoolingdir receives FileA, after it is picked up by Flume and renamed to FileA.COMPLETED placing another file of the same original name (FileA) will cause Flume to log an IllegalStateException indefinitely. This is likely due to Flume attempting to rename the second FileA to FileA.COMPLETED, but finding that the file already exists.

      When Flume has entered this state, it can only be recovered by removing the .COMPLETED file from the directory and restarting the agent.

      Log message looks like this:

      02 Jul 2013 21:32:09,371 ERROR [pool-4-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:164) - Uncaught exception in Runnable
      java.lang.IllegalStateException: Serializer has been closed
      at org.apache.flume.serialization.LineDeserializer.ensureOpen(LineDeserializer.java:124)
      at org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:88)
      at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221)
      at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

        Issue Links

          Activity

          Jonathan Cooper-Ellis created issue -
          Jonathan Cooper-Ellis made changes -
          Field Original Value New Value
          Description If a spoolingdir receives FileA, after it is picked up by Flume and renamed to FileA.COMPLETED placing another file of the same original name (FileA) will cause Flume to log an IllegalStateException indefinitely. This is likely due to Flume attempting to rename the second FileA to FileA.COMPLETED, but finding that the file already exists.

          When Flume has entered this state, it can only be recovered by removing the .COMPLETED file from the directory and restarting the agent.
          If a spoolingdir receives FileA, after it is picked up by Flume and renamed to FileA.COMPLETED placing another file of the same original name (FileA) will cause Flume to log an IllegalStateException indefinitely. This is likely due to Flume attempting to rename the second FileA to FileA.COMPLETED, but finding that the file already exists.

          When Flume has entered this state, it can only be recovered by removing the .COMPLETED file from the directory and restarting the agent.

          Log message looks like this:

          02 Jul 2013 21:32:09,371 ERROR [pool-4-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:164) - Uncaught exception in Runnable
          java.lang.IllegalStateException: Serializer has been closed
                  at org.apache.flume.serialization.LineDeserializer.ensureOpen(LineDeserializer.java:124)
                  at org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:88)
                  at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221)
                  at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                  at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  at java.lang.Thread.run(Thread.java:662)
          Phil Scala made changes -
          Link This issue Is contained by FLUME-2066 [ FLUME-2066 ]
          Phil Scala made changes -
          Assignee Phil Scala [ scaph01 ]
          Phil Scala made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Phil Scala made changes -
          Attachment FLUME-2119-0.patch [ 12608784 ]
          Phil Scala made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Affects Version/s v1.4.0 [ 12323372 ]
          Fix Version/s v1.5.0 [ 12324642 ]
          Phil Scala made changes -
          Attachment FLUME-2119-0.patch [ 12608784 ]
          Phil Scala made changes -
          Attachment FLUME-2119-0.patch [ 12608805 ]
          Phil Scala made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hari Shreedharan made changes -
          Fix Version/s v1.6.0 [ 12327047 ]
          Fix Version/s v1.5.0 [ 12324642 ]
          Johny Rufus made changes -
          Fix Version/s v1.6.0 [ 12327047 ]

            People

            • Assignee:
              Phil Scala
              Reporter:
              Jonathan Cooper-Ellis
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development