[FLUME-252] Update Tail to get rid of races and truncation problems.

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.0, 0.9.1
• Fix Version/s: 0.9.2
• Component/s: Sinks+Sources
• Assignee: Jonathan Hsieh
• Reporter: Jonathan Hsieh
• Created: 29/Sep/10 04:38
• Updated: 01/Apr/14 10:39
• Resolved: 04/Nov/10 21:02

Description

The first tail implementation used buffered readers and file readers. This caused problems because the read call was blocking and couldn't shutdown properly.

The second tail implementation was mroe closely based on gnu tail's C implementation but relies on RandomAccessFile. This version had problems with races (restarting from beginning FLUME-218) and truncation (new test in FLUME-218 patch by Eric Sammer).

The new approach will likely use NIO and nonblocking IO to act cleanly, or possibly use a JNI based approach to get to unix system calls to get and follow file descriptors or inode numbers.

Issue Links

• blocks: FLUME-258 TailDir should include source file name metadata for sink
• is duplicated by: FLUME-518 Flume is consistently using 70%+ of the CPU
• is related to: FLUME-320 Race condition in new NIO based TailSource
• is related to: FLUME-232 TailSource will add newlines in the middle of log lines
• relates to: FLUME-148 text source and tail source are incompatible with unix named pipes
• relates to: FLUME-2354 Fix possible race condition in Cursor
• relates to: FLUME-218 race condition in tailsource
• relates to: FLUME-205 TailSource reads lines using a method(readLine) which does character set interpretation and that breaks all my UTF-8 characters
• relates to: FLUME-248 TailSource not closing files