Flume
  1. Flume
  2. FLUME-1887

SpoolingDirectorySource throwing IllegalStateException while trying to rename/close stream

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.3.1
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      I am using SpoolDirectorySource to retrieve a 700MB file.

      Data is sent to Flume but at the end, when trying to rename file (.COMPLETED) and close stream, I get this exception :
      01 Feb 2013 01:18:40,579 INFO [pool-7-thread-1] (org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229) - Preparing to move file /Probes/1/MyFile.txt to /Probes/1/MyFile.txt.COMPLETED
      01 Feb 2013 01:18:40,580 ERROR [pool-7-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) - Uncaught exception in Runnable
      java.lang.IllegalStateException: File has been modified since being read: /Probes/1/MyFile.txt
      at org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:237)
      at org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:185)
      at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)
      01 Feb 2013 01:18:41,081 ERROR [pool-7-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) - Uncaught exception in Runnable
      java.io.IOException: Stream closed
      at java.io.BufferedReader.ensureOpen(BufferedReader.java:114)
      at java.io.BufferedReader.readLine(BufferedReader.java:309)
      at java.io.BufferedReader.readLine(BufferedReader.java:379)
      at org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180)
      at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)
      01 Feb 2013 01:18:41,582 ERROR [pool-7-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) - Uncaught exception in Runnable

      And this exception is thrown indefinitely, the file is never renamed.

      The same configuration is OK with smaller files (2MB):

      agent.channels = channel1
      agent.channels.channel1.type = memory
      agent.channels.channel1.capacity = 100000
      agent.channels.channel1.transactionCapacity = 1000

      agent.sources = source1
      agent.sources.source1.channels = channel1
      agent.sources.source1.type = spooldir
      agent.sources.source1.spoolDir = /Probes/1/
      agent.sources.source1.fileHeader = true
      agent.sources.source1.fileHeaderKey = fileName
      agent.sources.source1.batchSize = 100

      agent.sinks = sink1
      agent.sinks.sink1.channel = channel1
      agent.sinks.sink1.type = avro
      agent.sinks.sink1.hostname = 33.33.33.33
      agent.sinks.sink1.port = 5141

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Rudy Krol
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development