Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2525

flume should handle a zero byte .flumespool-main.meta file for the spooldir source

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0.1
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      When a zero byte .flumespool-main.meta file exists in the trackerDir (usually do to the partition filling up), flume will throw the following ambiguous error message when trying to read in new spool files:
      2014-10-19 18:28:31,333 ERROR org.apache.flume.client.avro.ReliableSpoolingFileEventReader: Exception opening file: /home/spooldir/input.log
      java.io.IOException: Not a data file.
      at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:102)
      at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97)
      at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:160)
      at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:149)
      at org.apache.flume.serialization.DurablePositionTracker.<init>(DurablePositionTracker.java:141)
      at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:76)
      at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:420)
      at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:215)
      at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:182)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

      Restarts of the flume agent do not resolve the issue. Only when the zero byte file is removed, will flume properly start processing files from the spooldir again.

        Attachments

        1. FLUME-2525.patch
          4 kB
          Johny Rufus
        2. FLUME-2525-1.patch
          2 kB
          Johny Rufus

          Activity

            People

            • Assignee:
              jrufus Johny Rufus
              Reporter:
              pdvorak Patrick Dvorak
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: