[FLUME-2829] Taildir Source doesn't populate Filename

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.0
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Andre F de Miranda
• Created: 27/Oct/15 03:04
• Updated: 27/Oct/15 08:19

Description

JIRA issue FLUME-2498 mentions taildir source should be able to populate Filename key in the header, however, this doesn't seem to be available yet.

Moreover, given the near certainty of taildir sources being used to tail dynamic filenames such as $hostname-$year-$month-$day.log it would be great to expose the regex groups to the generated headers in a fashion similar to the Regex Extractor Interceptor:

a1.sources.r1.filegroups.f1 = /path/to/my/logfile/([^-]+)-([^-]+)-([^-]+)-([^-]+).log
a1.sources.r1.filegroups.f1.regexgroups = s1 s2 s3 s4
a1.sources.r1.filegroups.f1.regexgroups.s1.header = hostname
a1.sources.r1.filegroups.f1.regexgroups.s2.header = year
a1.sources.r1.filegroups.f1.regexgroups.s3.header = month
a1.sources.r1.filegroups.f1.regexgroups.s4.header = day