[FLUME-2808] TailDirSource fails to process the source if one of the directories configured for file groups does not exist

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.0
• Assignee: bimal tandel
• Reporter: Arpit Gupta
• Created: 05/Oct/15 20:54
• Updated: 03/Feb/16 19:09

Description

Configured TailDirSource with multiple file groups. Some of the file group directories do not exist on all nodes on the cluster.

One the nodes where the file group directory did not exist TailDirSource failed to run with exception

05 Oct 2015 20:39:56,433 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: /dir/flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.IllegalStateException: Directory does not exist: /dir/log/hive
        at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:86)
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:55)
        at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:343)
        at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:91)
        at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:74)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

I built flume from sha: bd80c5e67deda4ec146f19fc92ff0c3bff61a982