[FLUME-431] Add attribute with dir path to events produced by tailing sources

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 0.9.4
• Fix Version/s: 0.9.5
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Alex Baranau
• Created: 27/Dec/10 08:54
• Updated: 05/Nov/14 09:07
• Resolved: 05/Nov/14 09:07

Description

Currently tailing sources (tail, taildir) add attribute with filename to event. Looks like we should put dir names too, since this can be critical info for event's processing logic.

E.g. tailing collectD output - it outputs stats data into dirs like /ubuntu904desktop.localdomain/cpu-0, /ubuntu904desktop.localdomain/cpu-1, etc. Here, path info is important for logs processing logic and should be available too.

Jon's suggestion:

My suggestion is to use a kwarg to make recording the dir a open, and to give the user option to pick attribute name.

So something like this:
tailDir("/dir",".*", 0, dirAttr="tailfiledir", fileAttr="tailfilename")

So a file /dir/file would produce events with metadata like:
event

{ taildirfile="/dir", tailfilename="file" }

Issue Links

• is blocked by: FLUME-410 Add ability to tail subdirs to tailDir sink