chenshangan - I like the idea but it is not possible to add a new method to the EventSerializer interface as it affects not just the built-in serializers, since it is not binary compatible with older code. There are custom serializers out there that would break if the interface is changed. If possible, can you try to do this without the change in the serializer interface.

Hari Shreedharan - What do you think if I add a new interface named BatchEventSerializer that extends the EventSerializer interface and add a new method in it ? I also deal with the interface HDFSWriter in the same way.

That sounds good. You'd need to support both the interfaces though, so you'd need to do some reflection.

Do you have any specific data to prove that adding events as one batch helps? I believe compression codecs actually hold events in a buffer and compress only when a flush is called or the buffer is full.

batchSize:200000

append in batch

append one by one

indicator explain:
all: overall time processing a batch
take: time cost in taking events from channel
append: time cost in append op
sync:time cost in flush

append op time significantly decreases, but the take time increases, I don't know why.
Another experiment is that send a large file and calculate the whole time, batch-append only cost half the time of one-by-one.

Hari Shreedharan I've updated the patch with testcase

This seems like a good idea. I did a quick review and it looks good. Since the serializer is the same for the life of the sink, we don't need to do an instanceOf check every time we write an event. We only need to do it once and reuse this info. We should fix that.

Also, batch size of 200000 is not realistic. I'd like to see if batch sizes between 1000 and 10000 show any difference with events of 500 bytes or so.

In our case, one batch consists of above 100 categories, each category has an independent bucketWriter, so actually one bucketWriter takes about 2000 messages in a batch.