This is the proposal of implementing a new tailing source.
This source watches the specified files, and tails them in nearly real-time once appends are detected to these files.
- This source is reliable and will not miss data even when the tailing files rotate.
- It periodically writes the last read position of each file in a position file using the JSON format.
- If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file.
- It can add event headers to each tailing file group.
A attached patch includes a config documentation of this.
This source requires Unix-style file system and Java 1.7 or later.