[FLUME-2502] Spool source's directory listing is inefficient - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.5.0
Fix Version/s: 1.6.0
Component/s: Sinks+Sources
Labels:
None

Description

As mentioned in FLUME-2309, the directory listing can it self become the bottleneck when accessing directories with a large number of files (>1M). The fix in that JIRA added in the ability to specify `RANDOM` as a Consume-Order to avoid sorting large lists.

The slowness of the directory listing is still un-addressed.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

FLUME-2502-0.patch
14/Oct/14 18:20
7 kB
Prateek Rungta
FLUME-2502-1.patch
14/Oct/14 18:59
3 kB
Prateek Rungta
FLUME-2502-2.patch
15/Oct/14 22:48
3 kB
Prateek Rungta
FLUME-2502-3.patch
15/Oct/14 23:08
3 kB
Prateek Rungta
FLUME-2502-final.patch
16/Oct/14 04:31
8 kB
Hari Shreedharan

Issue Links

relates to

FLUME-2505 Test added in FLUME-2502 is flaky

Resolved

Activity

People

Assignee:: Prateek Rungta

Reporter:: Prateek Rungta

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 14/Oct/14 18:09

Updated:: 17/Apr/15 01:21

Resolved:: 16/Oct/14 04:33