[FLUME-2918] TaildirSource is underperforming with huge parent directories - ASF JIRA

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.6.0
Fix Version/s: 1.7.0
Component/s: Sinks+Sources
Labels:
- performance

Release Note:
introduced an option in flume configuration for TAILDIR source to cache pattern matched files for huge directories
Flags:

Patch

Description

TailDir source cause high cpu utilization, when large amount of file is sitting in the target directory. File pattern matches only a single file, but the parent directory contains about 50,000 other file.

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending

Attachments

profiling_before.png
31/May/16 19:23
515 kB
Attila Simon
profiling_after.png
31/May/16 19:26
183 kB
Attila Simon
perftest.png
01/Jun/16 06:02
311 kB
Attila Simon
PerfHugeDir.java
01/Jun/16 06:02
6 kB
Attila Simon
test.csv
01/Jun/16 06:03
18 kB
Attila Simon
FLUME-2918-2.patch
02/Jun/16 03:15
31 kB
Attila Simon

Issue Links

relates to

FLUME-2934 Document new cachePatternMatching option for TaildirSource

Resolved

links to

Code review

Activity

People

Assignee:

Attila Simon

Reporter:

Attila Simon

Votes:

0 Vote for this issue

Watchers:

4 Start watching this issue

Dates

Created:

31/May/16 19:11

Updated:

20/Jun/16 10:20

Resolved:

20/Jun/16 08:58