Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2309

Spooling directory should not always consume the oldest file first.

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.5.0
    • Component/s: None
    • Labels:
    • Release Note:
      Adds consume order feature in the spooling directory source to allow files to be consumed in the order of oldest(default), youngest or random.

      Description

      The ReliableSpoolingFileEventReader reads the oldest file in the spooling directory first. This is done by listing the directory contents and then sorting file list based on timestamp. This may be very slow if there are a lot of files (of the order of 100K or more) in the directory.
      However, this is not always needed, there can be simple cases in which the order to consume the file is not important.
      There should be an option of consuming the files in arbitrary order, allowing the files to be consumed quickly without any delay.

        Attachments

        1. FLUME-2309-0.patch
          29 kB
          Muhammad Ehsan ul Haque
        2. FLUME-2309-0.patch
          28 kB
          Muhammad Ehsan ul Haque
        3. FLUME-2309-1.patch
          28 kB
          Muhammad Ehsan ul Haque
        4. FLUME-2309-commit.patch
          27 kB
          Hari Shreedharan

          Activity

            People

            • Assignee:
              ehsan Muhammad Ehsan ul Haque
              Reporter:
              ehsan Muhammad Ehsan ul Haque
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: