[FLUME-682] Improve SeqfileEventSink performance

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.4
• Fix Version/s: 0.9.5
• Component/s: Sinks+Sources
• Assignee: Disabled imported user
• Reporter: Disabled imported user
• Created: 24/Jun/11 20:04
• Updated: 29/Jun/11 00:08
• Resolved: 29/Jun/11 00:08

Description

The SeqfileEventSink disables output buffering in exchange for durability, however due to the async relationship between the source and sink, no such guarantees are possible. When using the SyslogTcpSourceThreads which enlists an internal buffer to store messages before they are read, a server crash would result in all buffered messages being lost. The lack of output buffering makes this much more likely to happen.

See previous discussion on flume-dev list: http://bit.ly/jQ1Gsv