Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2525

flume should handle a zero byte .flumespool-main.meta file for the spooldir source

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.0.1
    • 1.6.0
    • Sinks+Sources
    • None

    Description

      When a zero byte .flumespool-main.meta file exists in the trackerDir (usually do to the partition filling up), flume will throw the following ambiguous error message when trying to read in new spool files:
      2014-10-19 18:28:31,333 ERROR org.apache.flume.client.avro.ReliableSpoolingFileEventReader: Exception opening file: /home/spooldir/input.log
      java.io.IOException: Not a data file.
      at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:102)
      at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97)
      at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:160)
      at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:149)
      at org.apache.flume.serialization.DurablePositionTracker.<init>(DurablePositionTracker.java:141)
      at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:76)
      at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:420)
      at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:215)
      at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:182)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

      Restarts of the flume agent do not resolve the issue. Only when the zero byte file is removed, will flume properly start processing files from the spooldir again.

      Attachments

        1. FLUME-2525-1.patch
          2 kB
          Johny Rufus
        2. FLUME-2525.patch
          4 kB
          Johny Rufus

        Activity

          People

            jrufus Johny Rufus
            pdvorak Patrick Dvorak
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: