[FLUME-2918] TaildirSource is underperforming with huge parent directories

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.7.0
• Component/s: Sinks+Sources
• Labels: performance
• Release Note: introduced an option in flume configuration for TAILDIR source to cache pattern matched files for huge directories
• Flags: Patch
• Assignee: Attila Simon
• Reporter: Attila Simon
• Created: 31/May/16 19:11
• Updated: 20/Jun/16 10:20
• Resolved: 20/Jun/16 08:58

Description

TailDir source cause high cpu utilization, when large amount of file is sitting in the target directory. File pattern matches only a single file, but the parent directory contains about 50,000 other file.

Issue Links

• relates to: FLUME-2934 Document new cachePatternMatching option for TaildirSource
• links to: Code review