Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2955

Add file path to the header in TaildirSource

    Details

      Description

      After using TaildirSource to collect log data, I found that there is no file path in the flume event.
      Though TaildirSource supports add customed header value like this : a1.sources.r1.headers.f1.headerKey1 = value1,
      we can't add file path to the header when the filegroup path contains regex expression.File path is necessary to locate the log data.

      1. FLUME-2955_2.patch
        10 kB
        tinawenqiao
      2. FLUME-2955_3.patch
        10 kB
        tinawenqiao
      3. FLUME-2955_4.patch
        10 kB
        tinawenqiao
      4. FLUME-2955.patch
        10 kB
        tinawenqiao

        Issue Links

          Activity

          Hide
          wenqiao tinawenqiao added a comment -

          Make the pathHeader be configured. Default value is false.

          Show
          wenqiao tinawenqiao added a comment - Make the pathHeader be configured. Default value is false.
          Hide
          wenqiao tinawenqiao added a comment -

          Make the pathHeader be configured. Default value is false.

          Show
          wenqiao tinawenqiao added a comment - Make the pathHeader be configured. Default value is false.
          Hide
          denes Denes Arvay added a comment -

          hi tinawenqiao, could you please create a review request for this issue on https://reviews.apache.org for us to be able to review? Thanks.

          Show
          denes Denes Arvay added a comment - hi tinawenqiao , could you please create a review request for this issue on https://reviews.apache.org for us to be able to review? Thanks.
          Hide
          wenqiao tinawenqiao added a comment -

          The review request has been created. The link is https://reviews.apache.org/r/50134.

          Show
          wenqiao tinawenqiao added a comment - The review request has been created. The link is https://reviews.apache.org/r/50134 .
          Hide
          wenqiao tinawenqiao added a comment -

          According to Denes Arvay's suggestion, add fileHeader and fileHeaderKey parameters like SpoolDirectorySource.

          Show
          wenqiao tinawenqiao added a comment - According to Denes Arvay's suggestion, add fileHeader and fileHeaderKey parameters like SpoolDirectorySource.
          Hide
          wenqiao tinawenqiao added a comment -

          Improve the performance of two for cycles in ReliableTaildirEventReader.java

          Show
          wenqiao tinawenqiao added a comment - Improve the performance of two for cycles in ReliableTaildirEventReader.java
          Hide
          mpercy Mike Percy added a comment -

          +1. I am about to commit this.

          Show
          mpercy Mike Percy added a comment - +1. I am about to commit this.
          Hide
          mpercy Mike Percy added a comment -

          (I am making a very minor formatting edit to a test on commit)

          Show
          mpercy Mike Percy added a comment - (I am making a very minor formatting edit to a test on commit)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1ca0765aae795a41a43e39324f5f1c8bae57b751 in flume's branch refs/heads/trunk from tinawenqiao
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=1ca0765 ]

          FLUME-2955. Add file path to the header in TaildirSource

          Allow for adding a file path to the header dynamically. This is
          particularly useful when the filegroup path contains a regex expression.

          (tinawenqiao via Mike Percy)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1ca0765aae795a41a43e39324f5f1c8bae57b751 in flume's branch refs/heads/trunk from tinawenqiao [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=1ca0765 ] FLUME-2955 . Add file path to the header in TaildirSource Allow for adding a file path to the header dynamically. This is particularly useful when the filegroup path contains a regex expression. (tinawenqiao via Mike Percy)
          Hide
          mpercy Mike Percy added a comment -

          Pushed to trunk. Thank you for the patch, tinawenqiao!

          Show
          mpercy Mike Percy added a comment - Pushed to trunk. Thank you for the patch, tinawenqiao !
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in Flume-trunk-hbase-1 #183 (See https://builds.apache.org/job/Flume-trunk-hbase-1/183/)
          FLUME-2955. Add file path to the header in TaildirSource (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=1ca0765aae795a41a43e39324f5f1c8bae57b751)

          • flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java
          • flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java
          • flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSourceConfigurationConstants.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSource.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-1 #183 (See https://builds.apache.org/job/Flume-trunk-hbase-1/183/ ) FLUME-2955 . Add file path to the header in TaildirSource (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=1ca0765aae795a41a43e39324f5f1c8bae57b751 ) flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSourceConfigurationConstants.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSource.java

            People

            • Assignee:
              wenqiao tinawenqiao
              Reporter:
              wenqiao tinawenqiao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development