This patch makes the poolDelay property configurable.
The default value of 500ms is retained.

Thanks Somin, the idea seems reasonable to me, but just could you elaborate on the issue you were seeing that this fixes. Where I am going is: while 500ms is fast for checking for new files, if you were seeing race conditions where the file was still being writen to when this timer realized there was a new file then you could still have this issue, just not as frequently. We may need to focus on how the file was placed into the spool directory.

Thanks for your interest in this issue. The idea is to make the poll delay configurable, so users can set it as per their application. If new files will be posted only once every hour, then polling for new files twice a second can be waste of resource and also the log file will grow large due to log message is generated twice a second. This has nothing to do with flume seeing the file before its written completely. For that issue we can make use of excludes in file names. This patch is meant to enhance the configuration Flume provides now.
In my opinion, the poll delay should have been configurable in the first place. Do you see why this isn't the case.

+1, the use case seems reasonable

Thanks Somin, yep, your explanation makes sense.

Phil Scala and Johny Rufus - can you review the attached patch and provide your comments. I have given a link to the review board in links section.

Johny Rufus can u commit this if it looks good ?

Roshan Naik, the changes look good, the patch doesnt apply cleanly, had to rebase, will commit this as soon as I run the tests

Commit af63d38fada97a06c542ad875ef31ea3e74d53cc in flume's branch refs/heads/trunk from Johny Rufus
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=af63d38 ]

FLUME-2704. Configurable poll delay for spooling directory source

(Somin Mithraa via Johny Rufus)

Commit a6b55f18350cb458f8eb23b95c1cff0812960673 in flume's branch refs/heads/flume-1.7 from Johny Rufus
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=a6b55f1 ]

FLUME-2704. Configurable poll delay for spooling directory source

(Somin Mithraa via Johny Rufus)

Thanks Somin Mithraa for the patch !

UNSTABLE: Integrated in Flume-trunk-hbase-1 #147 (See https://builds.apache.org/job/Flume-trunk-hbase-1/147/)
FLUME-2704. Configurable poll delay for spooling directory source (johnyrufus: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=af63d38fada97a06c542ad875ef31ea3e74d53cc)

• flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
• flume-ng-doc/sphinx/FlumeUserGuide.rst
• flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java