Flume
  1. Flume
  2. FLUME-1899

Make SpoolDir work with Sub-Directories

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: v1.4.0, v1.3.1
    • Fix Version/s: v1.6.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      SpoolrDir currently monitors a directory and can not handle sub-directories. This JIRA is to make SpoolDir able to walk down a source directory and monitor new files.

        Activity

        Hide
        Hari Shreedharan added a comment -

        I updated the changelog.

        Show
        Hari Shreedharan added a comment - I updated the changelog.
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-98 #2 (See https://builds.apache.org/job/Flume-trunk-hbase-98/2/ ) FLUME-2408 . Remove FLUME-1899 from CHANGELOG. (harishreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=d0d00c3711c88c4da942dafb34ef3ab8861afdcd ) CHANGELOG
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - FAILURE: Integrated in flume-trunk #642 (See https://builds.apache.org/job/flume-trunk/642/ ) FLUME-2408 . Remove FLUME-1899 from CHANGELOG. (harishreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=d0d00c3711c88c4da942dafb34ef3ab8861afdcd ) CHANGELOG
        Hide
        ASF subversion and git services added a comment -

        Commit 4c5b602c5d65b66d2870dd5567d20c167e41dbaf in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4c5b602 ]

        FLUME-2408. Remove FLUME-1899 from CHANGELOG.

        Show
        ASF subversion and git services added a comment - Commit 4c5b602c5d65b66d2870dd5567d20c167e41dbaf in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4c5b602 ] FLUME-2408 . Remove FLUME-1899 from CHANGELOG.
        Hide
        ASF subversion and git services added a comment -

        Commit d5fad3b00083487204bd27cc07a1a724ba1053d0 in flume's branch refs/heads/flume-1.5 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d5fad3b ]

        FLUME-2408. Remove FLUME-1899 from CHANGELOG.

        Show
        ASF subversion and git services added a comment - Commit d5fad3b00083487204bd27cc07a1a724ba1053d0 in flume's branch refs/heads/flume-1.5 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d5fad3b ] FLUME-2408 . Remove FLUME-1899 from CHANGELOG.
        Hide
        ASF subversion and git services added a comment -

        Commit d0d00c3711c88c4da942dafb34ef3ab8861afdcd in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d0d00c3 ]

        FLUME-2408. Remove FLUME-1899 from CHANGELOG.

        Show
        ASF subversion and git services added a comment - Commit d0d00c3711c88c4da942dafb34ef3ab8861afdcd in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d0d00c3 ] FLUME-2408 . Remove FLUME-1899 from CHANGELOG.
        Hide
        Vish added a comment -

        In release document https://flume.apache.org/releases/1.5.0.html this issue is listed under improvement section
        But there is no mention of property name: recursiveDirectorySearch in 1.5.0 user guide and using this property with 1.5.0 does not work
        are we having this improvement in 1.6.0, if so can we update the release notes for 1.5.0

        Show
        Vish added a comment - In release document https://flume.apache.org/releases/1.5.0.html this issue is listed under improvement section But there is no mention of property name: recursiveDirectorySearch in 1.5.0 user guide and using this property with 1.5.0 does not work are we having this improvement in 1.6.0, if so can we update the release notes for 1.5.0
        Hide
        Phil Scala added a comment - - edited

        Attached is my first patch for this issue (the patch is against the v1.4 code base). I will continue to run some more tests.

        The patch details:

        • A new property has been added to the SpoolDirectorySource (property name: recursiveDirectorySearch)
        • the default for this setting is "false"...meaning its not enabled by default so there are no wierd side effects when moving from 1.3 to 1.4 flume agents.
        • when going through directories, ignored are:
          • directories starting with "." ...
          • any directories that match the ignore pattern (same pattern as the file ignore pattern)
        • 2 new unit tests were created for this feature.
        • FlumeUserGuide updated with this new property

        I waffled on one change and will chat over the dev mailing list...

        the FileFilter was an anonymous instance created inline, my changes made it double in size, so I made the filter a final instance member...comments on that welcome

        Show
        Phil Scala added a comment - - edited Attached is my first patch for this issue (the patch is against the v1.4 code base). I will continue to run some more tests. The patch details: A new property has been added to the SpoolDirectorySource (property name: recursiveDirectorySearch) the default for this setting is "false"...meaning its not enabled by default so there are no wierd side effects when moving from 1.3 to 1.4 flume agents. when going through directories, ignored are: directories starting with "." ... any directories that match the ignore pattern (same pattern as the file ignore pattern) 2 new unit tests were created for this feature. FlumeUserGuide updated with this new property I waffled on one change and will chat over the dev mailing list... the FileFilter was an anonymous instance created inline, my changes made it double in size, so I made the filter a final instance member...comments on that welcome
        Hide
        Phil Scala added a comment -

        Implementation locally into the 1.4 code base is nearly completed, unit tests and user guide document updated ..patch will be forthcoming post some last checks and tests

        Show
        Phil Scala added a comment - Implementation locally into the 1.4 code base is nearly completed, unit tests and user guide document updated ..patch will be forthcoming post some last checks and tests

          People

          • Assignee:
            Phil Scala
            Reporter:
            Tzur Turkenitz
          • Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development