Chris and Karthik,
Thank you for your sharing your experience and thinking. These are very useful for me.
ShuffleHandler is an auxiliary service loaded in the NodeManager. It's shared across all containers.
I see. I have to redesign it to run combiner in container.
Carlo Curino and I experimented with this, but (a) saw only slight improvements in job performance and (b) the changes to the AM to accommodate a new task type were extensive.
This is very interesting. In fact, I prototyped to run combiner at the end of MapTask as the first version. And, its performance was good. In this case, I found that it's needed to add new status to MapTask because of assuring fault tolerance. Is it acceptable for hadoop to do that?
With logic to manage skew, we're hoping that scheduling an aggressive range can have a similar effect to combiner tasks, without introducing the new task type.
This seems to be good approach to deal with rack-level aggregation. Do you have some result to
1. Perform node-level aggregation (reduce) at the end of maps in co-ordination with AM.
2. Perform rack-level aggregation at the end of node-level aggregation again in co-ordination with AM. The aggregation could be performed in parallel across the involved nodes such that each node has aggregated values of different keys.
3. Schedule reducers taking the key-distribution into account across racks.
The con will be that the shuffle won't be asynchronous to map computation, but hopefully this wouldn't offset the gains of decreased network and disk I/O.
The balance between the gains by asynchronous processing and the one by decreasing network and disk I/O. In my previous experiment, it deeply depends on number of reducers. I think these gains are trade-off, so parameters are necessary to deal with various workloads.
PS. http://dl.acm.org/citation.cfm?id=1901088 documents the advantages of multi-level aggregation in the context of graph algorithms modeled as iterative MR jobs.
I'm going read it
It's time for me to create the new revision of design note with reflecting your opinion.