[FLUME-2352] HDFSCompressedDataStream should support appendBatch

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.5.0
• Component/s: Sinks+Sources
• Assignee: chenshangan
• Reporter: chenshangan
• Created: 01/Apr/14 03:59
• Updated: 04/May/15 21:49

Description

compressing events in batch is much more efficient than compressing one by one.
I set hdfs.batchSize to 200000, when I use appendBatch() in BucketWriter, the append operation cost less than 1 seconds, while one by one might cost 10 seconds.