[FLUME-2525] flume should handle a zero byte .flumespool-main.meta file for the spooldir source

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0.1
• Fix Version/s: 1.6.0
• Component/s: Sinks+Sources
• Assignee: Johny Rufus
• Reporter: Patrick Dvorak
• Created: 28/Oct/14 23:31
• Updated: 06/Nov/14 20:44
• Resolved: 06/Nov/14 19:33

Description

When a zero byte .flumespool-main.meta file exists in the trackerDir (usually do to the partition filling up), flume will throw the following ambiguous error message when trying to read in new spool files:
2014-10-19 18:28:31,333 ERROR org.apache.flume.client.avro.ReliableSpoolingFileEventReader: Exception opening file: /home/spooldir/input.log
java.io.IOException: Not a data file.
at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:102)
at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97)
at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:160)
at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:149)
at org.apache.flume.serialization.DurablePositionTracker.<init>(DurablePositionTracker.java:141)
at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:76)
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:420)
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:215)
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:182)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Restarts of the flume agent do not resolve the issue. Only when the zero byte file is removed, will flume properly start processing files from the spooldir again.