[MAPREDUCE-5649] Reduce cannot use more than 2G memory for the final merge

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.1, 2.8.0, 2.7.2, 3.0.0-alpha1
• Component/s: mrv2
• Labels: 2.6.1-candidate 2.7.2-candidate
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Gera Shegalov
• Reporter: stanley shi
• Created: 25/Nov/13 05:32
• Updated: 06/Jan/17 01:00
• Resolved: 04/May/15 19:05

Description

In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method:

int maxInMemReduce = (int)Math.min(
Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE);

This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts.

Issue Links

• is duplicated by: MAPREDUCE-4883 Reducer's Maximum Shuffle Buffer Size should be enlarged for 64bit JVM
• relates to: MAPREDUCE-4883 Reducer's Maximum Shuffle Buffer Size should be enlarged for 64bit JVM