Change TaskTracker.MapOutputServlet to send only the IFile segment, validate checksum in Reduce

• Key: HADOOP-4699
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Chris Douglas
• Reporter: Chris Douglas
• Fix Version/s: 0.20.0

Description

Instead of validating the checksum of the IFile segment in MapOutputServlet, validation may be left to the reduce. While failures may not be detected until late in the reduce, the throughput and CPU improvements should make up for it in the average case.

Comments

[Jothi Padmanabhan]

This makes sense. However, I think we should do a measurement of the performance overhead experimentally to validate the theory. If there is no significant overhead due to checksum validation, we might be better off doing it in the Servlet as well to help prevent transmission of the corrupted output. No?

[Jothi Padmanabhan]

Disregard my previous comment, if we use NIO, we would not be reading the file at the servlet all and so cannot validate the checksum.

[Chris Douglas]

Removes the checksum validation from MapOutputServlet

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12395171/4699-0.patch
against trunk revision 724229.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3688/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3688/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3688/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3688/console

This message is automatically generated.

[Chris Douglas]

HADOOP-4688 validates this patch. It wasn't committed before Hudson ran, but it passes on my machine.

[Jothi Padmanabhan]

Changes look fine.
I see that you have removed the Debug statements introduced by Arun for HADOOP-3647. Is it no longer required?
And curious, did you benchmark this patch against trunk for performance?

[Chris Douglas]

I see that you have removed the Debug statements introduced by Arun for HADOOP-3647. Is it no longer required?

HADOOP-4754 was going to address this directly, but it would have conflicted with other patches, so I abandoned it. If you think it merits a separate issue, then I can rework both patches.

And curious, did you benchmark this patch against trunk for performance?

No. It probably won't have a measurable impact.

[Jothi Padmanabhan]

If you think it merits a separate issue, then I can rework both patches.

If the problem underlying HADOOP-3647 is resolved, I am fine with just removing the debug statements here itself.

[Jothi Padmanabhan]

+1

[Chris Douglas]

I just committed this.