multiple spills/reducers test case for mergeParts() needed

• Key: HADOOP-4688
• Type: Test
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Chris Douglas
• Reporter: Yuri Pradkin
• Affects Version/s: 0.19.1
• Fix Version/s: 0.20.0

Description

The first patch version (incorrect) for http://issues.apache.org/jira/browse/HADOOP-4614 exposed a problem that existing tests do not exercise some code paths of MapOutputBuffer::mergeParts(). In particular, we need a test that would validate output when numSpills >1 and partitions >1.

This might be a test that has more than 1 reducer and forces multiple first level (map) spills by setting io.sort.mb (and possibly io.sort.spill.percent) to small values.

Comments

[Nigel Daley]

+1 to such a test!

[Hudson]

Integrated in Hadoop-trunk #668 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/668/)
. Update CHANGES.txt for moving this back to 0.18.3.

[Chris Douglas]

Modified MiniMRDFSSort to spill multiple times

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12395029/4688-0.patch
against trunk revision 723918.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3682/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3682/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3682/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3682/console

This message is automatically generated.

[Chris Douglas]

Neither test failure is related. (HADOOP-4716, Chukwa test timeout)

[Jothi Padmanabhan]

+1

[Chris Douglas]

I just committed this

[Hudson]

Integrated in Hadoop-trunk #683 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/683/)
. Modify the MiniMRDFSSort unit test to spill multiple times,
exercising the map-side merge code.