Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 1.7.0
-
Fix Version/s: 1.7.0
-
Component/s: Sinks+Sources
-
Labels:
Description
After using TaildirSource to collect log data, I found that there is no file path in the flume event.
Though TaildirSource supports add customed header value like this : a1.sources.r1.headers.f1.headerKey1 = value1,
we can't add file path to the header when the filegroup path contains regex expression.File path is necessary to locate the log data.
Issue Links
- links to
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Make the pathHeader be configured. Default value is false.
hi tinawenqiao, could you please create a review request for this issue on https://reviews.apache.org for us to be able to review? Thanks.
The review request has been created. The link is https://reviews.apache.org/r/50134.
According to Denes Arvay's suggestion, add fileHeader and fileHeaderKey parameters like SpoolDirectorySource.
Improve the performance of two for cycles in ReliableTaildirEventReader.java
+1. I am about to commit this.
(I am making a very minor formatting edit to a test on commit)
Commit 1ca0765aae795a41a43e39324f5f1c8bae57b751 in flume's branch refs/heads/trunk from tinawenqiao
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=1ca0765 ]
FLUME-2955. Add file path to the header in TaildirSource
Allow for adding a file path to the header dynamically. This is
particularly useful when the filegroup path contains a regex expression.
(tinawenqiao via Mike Percy)
Pushed to trunk. Thank you for the patch, tinawenqiao!
UNSTABLE: Integrated in Flume-trunk-hbase-1 #183 (See https://builds.apache.org/job/Flume-trunk-hbase-1/183/)
FLUME-2955. Add file path to the header in TaildirSource (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=1ca0765aae795a41a43e39324f5f1c8bae57b751)
- flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java
- flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java
- flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSourceConfigurationConstants.java
- flume-ng-doc/sphinx/FlumeUserGuide.rst
- flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirSource.java
Make the pathHeader be configured. Default value is false.