[FLUME-2918] TaildirSource is underperforming with huge parent directories - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.6.0
Fix Version/s: 1.7.0
Component/s: Sinks+Sources
Labels:
- performance

Release Note:
introduced an option in flume configuration for TAILDIR source to cache pattern matched files for huge directories
Flags:

Patch

Description

TailDir source cause high cpu utilization, when large amount of file is sitting in the target directory. File pattern matches only a single file, but the parent directory contains about 50,000 other file.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

test.csv
01/Jun/16 06:03
18 kB
Attila Simon
profiling_before.png
31/May/16 19:23
515 kB
Attila Simon
profiling_after.png
31/May/16 19:26
183 kB
Attila Simon
perftest.png
01/Jun/16 06:02
311 kB
Attila Simon
PerfHugeDir.java
01/Jun/16 06:02
6 kB
Attila Simon
FLUME-2918-2.patch
02/Jun/16 03:15
31 kB
Attila Simon

Issue Links

relates to

FLUME-2934 Document new cachePatternMatching option for TaildirSource

Resolved

links to

Code review

Activity

People

Assignee:: Attila Simon

Reporter:: Attila Simon

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 31/May/16 19:11

Updated:: 20/Jun/16 10:20

Resolved:: 20/Jun/16 08:58