Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2918

TaildirSource is underperforming with huge parent directories

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.7.0
    • Component/s: Sinks+Sources
    • Labels:
    • Release Note:
      introduced an option in flume configuration for TAILDIR source to cache pattern matched files for huge directories
    • Flags:
      Patch

      Description

      TailDir source cause high cpu utilization, when large amount of file is sitting in the target directory. File pattern matches only a single file, but the parent directory contains about 50,000 other file.

        Attachments

        1. FLUME-2918-2.patch
          31 kB
          Attila Simon
        2. PerfHugeDir.java
          6 kB
          Attila Simon
        3. perftest.png
          311 kB
          Attila Simon
        4. profiling_after.png
          183 kB
          Attila Simon
        5. profiling_before.png
          515 kB
          Attila Simon
        6. test.csv
          18 kB
          Attila Simon

          Issue Links

          relates to

          Documentation - Documentation or Website FLUME-2934 Document new cachePatternMatching option for TaildirSource

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link Code review

            Activity

              People

              • Assignee:
                sati Attila Simon
                Reporter:
                sati Attila Simon
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: