[FLUME-2344] New source for tailing files

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.4.0
• Component/s: Sinks+Sources

Environment

Centos 6.4, Java 1.6.0_34

Release Note

Patch has been uploaded.
It has been create a new source for tailing files in Flume.
If a file rotates more than once, you couldn't recover those data with this Source.
For example, if you're watching catalina.out and this file rotates to catalina.out.1 and Flume is down, when you turn Flume on, you read the data that you didn't read in catalina.out.1, but if while Flume is down there was another rotation (catalina.out.2), you couldn't recover those data.
If for some reason it's not enable to recover data, the source'll start to read the original file from the beginning.

To detect rotation and which file has to read, the Source keeps a file with file is watching, last offset we read and last time the rotated file was modified.

• Assignee: Unassigned
• Reporter: Guillermo Ortiz Fernández, Pragsis.
• Created: 10/Mar/14 12:51
• Updated: 25/Jun/16 06:42
• Resolved: 25/Jun/16 06:42

Description

New source to be able to tail a file. There's a extra file when it saves the last offset until it has been read. So, it Flume is down, it could read the data it lost. It has been implemented control about rotated files.

The possible variables to configure this source are:
-BufferSize gives us the possibility to send little to little.
-Separator to cut the lines wherever we want to to generate our events.
-WatchedFile to indicate what file we want to watch.
-RotatedFile to indicate where the wacthed file is going to rotate.
-Type of events to indicate if we generate one event per line or we want to group many lines and just emit one event.

Issue Links

• is related to: FLUME-2498 Implement Taildir Source