After checking the control flow it turned out that the function (ReliableTaildirEventReader.getMatchFiles) - which is responsible for checking whether new files has been added or removed within the parent dir of the file pattern - is called every time when the PollableSourceRunner$PollingRunner instructed the TaildirSource to harvest new data. Even though nothing changed in that directory. This check requires listing all of the files and filtering those using a pattern match and a isDirectory check within a single if statement calling directory check first. Profiling showed that isDirectory is much more expensive call than pattern match on the filename so changing the order of the expressions would speed up the evaluation(short-circuit nature of the java evaluation of boolean expressions) hence listing the dir. On the other hand caching what was the last modification time of the parent directory and the list of matched files for each filepattern prevent unnecessary rechecks.

21% of thread time is spent on ReliableTaildirEventReader.getMatches() -> File.listFiles() -> FileFilter.accept() -> File.isDirectory()

profiling_after.png shows that with the fix time spent on getMatchFiles() (same workload with occasionally (2 per min) adding new files to the directory) is reduced to 3.4% of thread time

Comparing how could the same functionality be implemented clarified that using java.nio.file.DirectoryStream to list the files gives the best overall performance (only very first invocation has a JIT overhead when it performs little bit worse than the proper FileFilter). Please see attachments.

• PerfHugeDir.java generated the execution times
• test.csv captured result of executing PerfHugeDir.main()
• perftest.png charted version of the csv data (execution time in millisecs comparing the different implementations)
  I started with a directory of 59k files, only a single file matched the pattern, there were couple of subdirs. After ~230 rounds I started massively removing the files not matched by the pattern and reduced the number to ~20 files all together within the parent dir which reduction was responsible for the fade out. (Secondly I ran the same test starting with empty dir and adding 300files/sec to 59k that was also won by DirectoryStream. No attachment for this.)

use java.nio.file.DirectoryStream to filter files
make pattern match calculation optionally cached
add junit tests
add javadoc
add licence
maven build passes

patch is considered final ready for review

Patch is available for review: https://reviews.apache.org/r/48161/

+1. I'm committing the latest version of the patch from review board.

Commit 7d1e683fbd7d261fff9fcf17ad78fd8469c64905 in flume's branch refs/heads/trunk from Mike Percy
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=7d1e683 ]

FLUME-2918. Speed up TaildirSource on directories with many files

This patch greatly improves the performance of TaildirSource on
directories that contain a large number of files.

(Attila Simon via Mike Percy)

Committed to trunk.

Thanks for the contribution, Attila!

FAILURE: Integrated in Flume-trunk-hbase-1 #164 (See https://builds.apache.org/job/Flume-trunk-hbase-1/164/)
FLUME-2918. Speed up TaildirSource on directories with many files (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=7d1e683fbd7d261fff9fcf17ad78fd8469c64905)

• flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TailFile.java
• flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirMatcher.java
• flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirMatcher.java
• flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSourceConfigurationConstants.java
• flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java
• flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSource.java
• flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java