What exactly needs to happen here ? Should they all be 100 ?

Yeah I think so. HDFSEventSink might have increased and a few other sources/sinks have had batch sizes added. I just think that we should standardize this across all sinks/sources. 100 seems good.

Maybe back in March it would have been a good time to change all the defaults batch size values. But now?

I see that the following sinks have their batch sizes at 100 ..

• Rolling File, HDFS, HBase, Async HBase, Null, Avro

The avro sink seems to be picking up its default in a relatively complicated manner from RpcClientConfigurationConstants.DEFAULT_BATCH_SIZE inside NettyAvroRpcClient.configure()

Following Sinks do not use batch size.. and its probably ok that way (they operate as if batch size = 1).

• IRC, Logger

The following do not have any default although the documentation claims otherwise :

• ElasticSearch

In ElasticSearch it defaults to 0. This appears to be a bug.

I dont see any other variation as such in the default batch sizes.

Set default batchSize to 100 in ElasticSearch sink

Setting default batch size to 100 for Elastic Search sink

Mike, didn't FLUME-1631 just increase the HDFSEventSink batch size to 100 from 1? The good news is that, thanks to Roshan's detective work, it looks like we only have to change the ElasticSearchSink.

Could somebody take a look at committing this patch. its a trivial one.

You guys are right, nice detective work Roshan.

+1

Pushed to trunk & flume-1.4 branches. Thanks for the patch Roshan!

Integrated in flume-trunk #347 (See https://builds.apache.org/job/flume-trunk/347/)
FLUME-1076. Sink default batch sizes vary wildly. (Revision 58173b8983027124a61783b4326dee3347ab7552)

Result = SUCCESS
mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=58173b8983027124a61783b4326dee3347ab7552
Files :

• flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java