[HADOOP-4688] multiple spills/reducers test case for mergeParts() needed

• Type: Test
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.19.1
• Fix Version/s: 0.20.0
• Hadoop Flags: Reviewed
• Assignee: Christopher Douglas
• Reporter: Yuri Pradkin
• Created: 19/Nov/08 19:01
• Updated: 08/Jul/09 16:53
• Resolved: 08/Dec/08 19:43

Description

The first patch version (incorrect) for http://issues.apache.org/jira/browse/HADOOP-4614 exposed a problem that existing tests do not exercise some code paths of MapOutputBuffer::mergeParts(). In particular, we need a test that would validate output when numSpills >1 and partitions >1.

This might be a test that has more than 1 reducer and forces multiple first level (map) spills by setting io.sort.mb (and possibly io.sort.spill.percent) to small values.