Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2502

Spool source's directory listing is inefficient

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.0
    • 1.6.0
    • Sinks+Sources
    • None

    Description

      As mentioned in FLUME-2309, the directory listing can it self become the bottleneck when accessing directories with a large number of files (>1M). The fix in that JIRA added in the ability to specify `RANDOM` as a Consume-Order to avoid sorting large lists.

      The slowness of the directory listing is still un-addressed.

      Attachments

        1. FLUME-2502-0.patch
          7 kB
          Prateek Rungta
        2. FLUME-2502-1.patch
          3 kB
          Prateek Rungta
        3. FLUME-2502-2.patch
          3 kB
          Prateek Rungta
        4. FLUME-2502-3.patch
          3 kB
          Prateek Rungta
        5. FLUME-2502-final.patch
          8 kB
          Hari Shreedharan

        Issue Links

        relates to

        Bug - A problem which impairs or prevents the functions of the product. FLUME-2505 Test added in FLUME-2502 is flaky

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            prateekrungta Prateek Rungta
            prateekrungta Prateek Rungta
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment