In ReduceTask::fetchOutputs, wait for result can be improved slightly

• Key: HADOOP-3427
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Devaraj Das
• Reporter: Devaraj Das
• Fix Version/s: 0.18.0

Description

The getCopyResult call in ReduceTask::fetchOutputs waits for 2 seconds if the results list is empty. This can be improved to wait only when sufficient number of fetches (above a certain threshold) have been scheduled. The threshold should ensure that all the fetcher threads would be busy enough, and in this case, the call to getCopyResult would return only when a fetcher thread wakes it up.

Comments

[Devaraj Das]

This patch mainly introduces this change - Defines a number to indicate when the shuffle scheduler should stop and just wait for notifications from the fetchers. This number is hardcoded to (4 * numCopiers).

[Devaraj Das]

This fixes a comment in the earlier patch

[Devaraj Das]

Retrying hudson

[Devaraj Das]

The last patch had an incorrect wait()

[Devaraj Das]

Trying to run hudson again.

[Mahadev konar]

+1 the patch looks good, though the fetchoutputs() needs refactoring urgently.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382819/3427.patch
against trunk revision 662913.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

-1 patch. The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2551/console

This message is automatically generated.

[Devaraj Das]

Retrying hudson

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382819/3427.patch
against trunk revision 663321.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2574/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2574/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2574/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2574/console

This message is automatically generated.

[Devaraj Das]

Cancelling patch due to conflicts with trunk

[Devaraj Das]

This is updated w.r.t the trunk.

[Devaraj Das]

queueing it for hudson. Forgot to thank Amareshwari for helping resolving the conflicts quickly.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383448/3427.3.patch
against trunk revision 663487.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2587/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2587/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2587/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2587/console

This message is automatically generated.

[Devaraj Das]

I just committed this.