Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2502

Spool source's directory listing is inefficient

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.0
    • 1.6.0
    • Sinks+Sources
    • None

    Description

      As mentioned in FLUME-2309, the directory listing can it self become the bottleneck when accessing directories with a large number of files (>1M). The fix in that JIRA added in the ability to specify `RANDOM` as a Consume-Order to avoid sorting large lists.

      The slowness of the directory listing is still un-addressed.

      Attachments

        1. FLUME-2502-0.patch
          7 kB
          Prateek Rungta
        2. FLUME-2502-1.patch
          3 kB
          Prateek Rungta
        3. FLUME-2502-2.patch
          3 kB
          Prateek Rungta
        4. FLUME-2502-3.patch
          3 kB
          Prateek Rungta
        5. FLUME-2502-final.patch
          8 kB
          Hari Shreedharan

        Issue Links

          Activity

            People

              prateekrungta Prateek Rungta
              prateekrungta Prateek Rungta
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: