Details
-
Type:
Improvement
-
Status: Patch Available
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: 1.5.0
-
Fix Version/s: None
-
Component/s: Sinks+Sources
-
Labels:None
Description
compressing events in batch is much more efficient than compressing one by one.
I set hdfs.batchSize to 200000, when I use appendBatch() in BucketWriter, the append operation cost less than 1 seconds, while one by one might cost 10 seconds.