[FLUME-2498] Implement Taildir Source

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Satoshi Iijima
• Created: 10/Oct/14 08:44
• Updated: 21/Sep/16 12:43
• Resolved: 18/Aug/15 02:56

Description

This is the proposal of implementing a new tailing source.
This source watches the specified files, and tails them in nearly real-time once appends are detected to these files.

• This source is reliable and will not miss data even when the tailing files rotate.
• It periodically writes the last read position of each file in a position file using the JSON format.
• If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file.
• It can add event headers to each tailing file group.

A attached patch includes a config documentation of this.
This source requires Unix-style file system and Java 1.7 or later.

Issue Links

• relates to: FLUME-2344 New source for tailing files
• links to: Review Board link