[MAPREDUCE-4502] Node-level aggregation with combining the result of maps

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha1
• Component/s: applicationmaster, mrv2
• Labels: BB2015-05-TBR
• Assignee: Tsuyoshi Ozawa
• Reporter: Tsuyoshi Ozawa
• Created: 01/Aug/12 06:46
• Updated: 12/May/16 18:23

Description

The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because the scope of combining is limited within only one MapTask. To solve this problem, it's a good way to aggregate the result of maps per node/rack by launch combiner.

This JIRA is to implement the multi-level aggregation infrastructure, including combining per container(MAPREDUCE-3902 is related), coordinating containers by application master without breaking fault tolerance of jobs.

Issue Links

• is related to: TAJO-374 Investigate more efficient intermediate shuffle methods

Sub-Tasks

• Combiner per node: Open
• Adding aggregationWaitMap for node-level combiner.: In Progress
• Adding new umbilical protocol RPC, "getAggregationTargets()", for node-level combiner.: In Progress
• Launching node-level combiner at the end stage of MapTask and ignoring aggregated inputs at ReduceTask: In Progress
• Adding AggregationWaitMap to some components(MRAppMaster, TaskAttemptListener, JobImpl, MapTaskImpl).: In Progress
• Add node-level aggregation flag feature(setNodeLevelAggregation(boolean)) to JobConf: Patch Available