[FLUME-2801] Performance improvement on TailDir source

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.7.0
• Component/s: Sinks+Sources
• Flags: Patch
• Assignee: Jun Seok Hong
• Reporter: Jun Seok Hong
• Created: 24/Sep/15 02:05
• Updated: 18/Jun/16 17:25
• Resolved: 17/Dec/15 18:48

Description

This a proposal of performance improvement for new tailing source FLUME-2498.
Taildir source reads a file by 1byte, so the performance is very low compared to tailing on exec source.
I tested lot's of ways to improve performance and implemented the best one.

Changes.

• Reading a file by a 8k block instead of 1 byte.
• Use byte[] for handling data instead of ByteArrayDataOutput/ByteBuffer(direct)/.. for better performance.
• Don't convert byte[] to string and vice verse.

Simple file reading test results.

File size: 100 MB,
Line size: 500 byte

Estimated time to read the file:

Reading 1byte(Using the code in Taildir)	32544 ms
Reading 8K Block	431 ms

Testing on flume, it catches up the performance of tailing on exec source. (30x performance boost)