[MAPREDUCE-268] Implement memory-to-memory merge in the reduce

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Arun Murthy
• Reporter: Arun Murthy
• Created: 14/May/09 07:31
• Updated: 05/Jun/12 02:37
• Resolved: 14/Sep/09 04:57

Description

HADOOP-3446 fixed the reduce to not flush the in-memory shuffled map-outputs before feeding to the reduce. However for latency-sensitive applications with lots of memory like the terasort this hurts performance since the fan-in for the final in-memory merge is too large (all 8000 map-outputs very in-memory) resulting in less than optimal performance.

When I put in an intermediate memory-to-memory merge for the terasort's reduce (there-by avoiding disk i/o) to cut the fan-in from 8000 to <100 the 'reduce' phase (including the local datanode-write) sped-up 250% (from 10s to 4s).

Issue Links

• is part of: MAPREDUCE-318 Refactor reduce shuffle code