[FLUME-2502] Spool source's directory listing is inefficient

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.6.0
• Component/s: Sinks+Sources
• Assignee: Prateek Rungta
• Reporter: Prateek Rungta
• Created: 14/Oct/14 18:09
• Updated: 17/Apr/15 01:21
• Resolved: 16/Oct/14 04:33

Description

As mentioned in FLUME-2309, the directory listing can it self become the bottleneck when accessing directories with a large number of files (>1M). The fix in that JIRA added in the ability to specify `RANDOM` as a Consume-Order to avoid sorting large lists.

The slowness of the directory listing is still un-addressed.

Issue Links

• relates to: FLUME-2505 Test added in FLUME-2502 is flaky