Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha, trunk
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:

      Description

      When set mapred.max.map.failures.percent and there does have some failed maps, then shuffle will hang

      1. MAPREDUCE-3927.patch
        0.9 kB
        MengWang
      2. MAPREDUCE-3927.patch
        0.8 kB
        MengWang
      3. MAPREDUCE-3927-1.patch
        7 kB
        Bhallamudi Venkata Siva Kamesh
      4. MAPREDUCE-3927-2.patch
        6 kB
        Bhallamudi Venkata Siva Kamesh
      5. MAPREDUCE-3927-3.patch
        6 kB
        Bhallamudi Venkata Siva Kamesh
      6. MAPREDUCE-3927-4.patch
        6 kB
        Jason Lowe

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12516178/MAPREDUCE-3927.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1938//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1938//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516178/MAPREDUCE-3927.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1938//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1938//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12516294/MAPREDUCE-3927.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1946//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1946//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516294/MAPREDUCE-3927.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1946//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1946//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        Just a quick look at the patch it looks good to me, but I really would like to see a unit test that can reproduce the issue. That way we know for sure that it is fixed, and that it does not get broken again in the future.

        Show
        Robert Joseph Evans added a comment - Just a quick look at the patch it looks good to me, but I really would like to see a unit test that can reproduce the issue. That way we know for sure that it is fixed, and that it does not get broken again in the future.
        Hide
        MengWang added a comment -

        ok i will write a unit test

        Show
        MengWang added a comment - ok i will write a unit test
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Shouldn't we add notifyAll() when the remaining maps is zero?

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Shouldn't we add notifyAll() when the remaining maps is zero?
        Hide
        Ravi Gummadi added a comment -

        copySucceeded() is updating the progress of map-tasks, but tipFailed() is not updating. My understanding is that overall progress of map tasks represents the completed maps' work. So it may be better to update it for the failed-map-tasks case also.

        Show
        Ravi Gummadi added a comment - copySucceeded() is updating the progress of map-tasks, but tipFailed() is not updating. My understanding is that overall progress of map tasks represents the completed maps' work. So it may be better to update it for the failed-map-tasks case also.
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Hi MengWang,
        Any update on the patch?

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Hi MengWang, Any update on the patch?
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Attaching patch for issue. Please review it.

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Attaching patch for issue. Please review it.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528430/MAPREDUCE-3927-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2406//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2406//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528430/MAPREDUCE-3927-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2406//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2406//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        Overall looks good. Couple of comments:

        In Job.java, this code:

        if (reportedAfterCompletion
            && status.getState() == JobStatus.State.SUCCEEDED) {
          mapProgress = mapProgress != 1.0f ? 1.0f : mapProgress;
          reduceProgress = reduceProgress != 1.0f ? 1.0f : reduceProgress;
        }
        

        would be better written as:

        if (reportedAfterCompletion
            && status.getState() == JobStatus.State.SUCCEEDED) {
          mapProgress = 1.0f;
          reduceProgress = 1.0f;
        }
        

        Also I noticed that when one or more map task fails (after max attempts have failed), the map completion percentage of the job isn't accurate until the job completes. (e.g.: reducer progress shows we're well past the shuffle stage even though map progress is <100%). If that were done properly, I'm not sure we'd need the SUCCEEDED == progress 1.0 fixup in Job.java.

        Show
        Jason Lowe added a comment - Overall looks good. Couple of comments: In Job.java, this code: if (reportedAfterCompletion && status.getState() == JobStatus.State.SUCCEEDED) { mapProgress = mapProgress != 1.0f ? 1.0f : mapProgress; reduceProgress = reduceProgress != 1.0f ? 1.0f : reduceProgress; } would be better written as: if (reportedAfterCompletion && status.getState() == JobStatus.State.SUCCEEDED) { mapProgress = 1.0f; reduceProgress = 1.0f; } Also I noticed that when one or more map task fails (after max attempts have failed), the map completion percentage of the job isn't accurate until the job completes. (e.g.: reducer progress shows we're well past the shuffle stage even though map progress is <100%). If that were done properly, I'm not sure we'd need the SUCCEEDED == progress 1.0 fixup in Job.java.
        Hide
        Devaraj K added a comment -

        @Kamesh : any update on this?

        Show
        Devaraj K added a comment - @Kamesh : any update on this?
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Cancelling the patch to address the comment

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Cancelling the patch to address the comment
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Hi Jason,
        Thanks for the review.

        Also I noticed that when one or more map task fails...

        Updated the patch. Please review

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Hi Jason, Thanks for the review. Also I noticed that when one or more map task fails... Updated the patch. Please review
        Hide
        Devaraj K added a comment -

        Thanks Kamesh for the patch.

        Source code changes looks good to me. I have tested this, it works fine after the patch.

        About the test, Can we have a test case which actually reproduces the issue mentioned in this defect and passes after this change?

        Show
        Devaraj K added a comment - Thanks Kamesh for the patch. Source code changes looks good to me. I have tested this, it works fine after the patch. About the test, Can we have a test case which actually reproduces the issue mentioned in this defect and passes after this change?
        Hide
        Jason Lowe added a comment -

        I'm not sure we should change the denominator of the progress calculation when map tasks fail. As I see it, the map percentage is a representation of how many map tasks have been processed regardless of successful or failure outcome. 50% map progress to me means half of the map tasks have been processed and half have yet to finish processing.

        For example, let's say we have 4 map tasks and are configured to allow at least one to fail. When 1 out of 4 have completed, we're sitting at 25% map completion. If another map task fails, I'd expect the completion to be 50% as we've processed two tasks and have yet to finish the other two. However the proposed change will report 33% completion since it will remove the failed task from the total task count instead of adding it to the processed task count. Adding it to the processed task count would also be in line with how a job reports completed maps, see JobImpl.getCompletedMaps().

        Show
        Jason Lowe added a comment - I'm not sure we should change the denominator of the progress calculation when map tasks fail. As I see it, the map percentage is a representation of how many map tasks have been processed regardless of successful or failure outcome. 50% map progress to me means half of the map tasks have been processed and half have yet to finish processing. For example, let's say we have 4 map tasks and are configured to allow at least one to fail. When 1 out of 4 have completed, we're sitting at 25% map completion. If another map task fails, I'd expect the completion to be 50% as we've processed two tasks and have yet to finish the other two. However the proposed change will report 33% completion since it will remove the failed task from the total task count instead of adding it to the processed task count. Adding it to the processed task count would also be in line with how a job reports completed maps, see JobImpl.getCompletedMaps().
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Thanks Deva and Jason for the reviews.

        @Jason
        I think, to get true progess the map/reduce phase,we should return 1f, when a map/reduce task is finished (SUCCEEDED or FAILED or KILLED). However, in the current code base, progress of the failed/killed map/reduce task will be < 1f.

        Proposed Change: JobImpl#computeProgress
        if (task.getType() == TaskType.MAP) {
          mapProgress += (task.isFinished() ? 1f : task.getProgress());
        } else {
          reduceProgress += (task.isFinished() ? 1f : task.getProgress());
        }
        
        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Thanks Deva and Jason for the reviews. @Jason I think , to get true progess the map/reduce phase,we should return 1f , when a map/reduce task is finished (SUCCEEDED or FAILED or KILLED). However, in the current code base, progress of the failed/killed map/reduce task will be < 1f . Proposed Change: JobImpl#computeProgress if (task.getType() == TaskType.MAP) { mapProgress += (task.isFinished() ? 1f : task.getProgress()); } else { reduceProgress += (task.isFinished() ? 1f : task.getProgress()); }
        Hide
        Jason Lowe added a comment -

        Looks like progress of a killed or failed task will always be zero, as TaskImpl.getProgress() returns 0 if there is no best attempt and TaskImpl.selectBestAttempt() skips all killed or failed attempts.

        Having the job interpret any finished task as 100% progressed should fix the job progress reporting for the case where there are failed tasks.

        Show
        Jason Lowe added a comment - Looks like progress of a killed or failed task will always be zero, as TaskImpl.getProgress() returns 0 if there is no best attempt and TaskImpl.selectBestAttempt() skips all killed or failed attempts. Having the job interpret any finished task as 100% progressed should fix the job progress reporting for the case where there are failed tasks.
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Please review the patch.

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Please review the patch.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12531070/MAPREDUCE-3927-3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2440//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2440//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531070/MAPREDUCE-3927-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2440//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2440//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        +1 (non-binding), looks good to me. Thanks Kamesh!

        Show
        Jason Lowe added a comment - +1 (non-binding), looks good to me. Thanks Kamesh!
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Any further comments? If not, can this be checked in?

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Any further comments? If not, can this be checked in?
        Hide
        Thomas Graves added a comment -

        code looks good. As stated above, it would be nice to have a test that tests the issue original described.

        Show
        Thomas Graves added a comment - code looks good. As stated above, it would be nice to have a test that tests the issue original described.
        Hide
        Jason Lowe added a comment -

        In the interest of getting this in quicker, updated Kamesh's patch so we explicitly test the waitUntilDone() method that was preventing the reducer from exiting after all map tasks had completed but some have failed.

        Show
        Jason Lowe added a comment - In the interest of getting this in quicker, updated Kamesh's patch so we explicitly test the waitUntilDone() method that was preventing the reducer from exiting after all map tasks had completed but some have failed.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12531447/MAPREDUCE-3927-4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2447//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2447//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531447/MAPREDUCE-3927-4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2447//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2447//console This message is automatically generated.
        Hide
        Thomas Graves added a comment -

        +1 lgtm. Thanks Kamesh and Jason!

        Show
        Thomas Graves added a comment - +1 lgtm. Thanks Kamesh and Jason!
        Hide
        Thomas Graves added a comment -

        committed to trunk, branch-2, and branch-0.23

        Show
        Thomas Graves added a comment - committed to trunk, branch-2, and branch-0.23
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2414 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2414/)
        MAPREDUCE-3927. Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2414 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2414/ ) MAPREDUCE-3927 . Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2341 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2341/)
        MAPREDUCE-3927. Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2341 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2341/ ) MAPREDUCE-3927 . Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2361 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2361/)
        MAPREDUCE-3927. Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2361 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2361/ ) MAPREDUCE-3927 . Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1074 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1074/)
        MAPREDUCE-3927. Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1074 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1074/ ) MAPREDUCE-3927 . Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #284 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/284/)
        merge -r 1348847:1348848 from branch-2. FIXES: MAPREDUCE-3927 (Revision 1348850)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348850
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #284 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/284/ ) merge -r 1348847:1348848 from branch-2. FIXES: MAPREDUCE-3927 (Revision 1348850) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348850 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1107/)
        MAPREDUCE-3927. Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846)

        Result = FAILURE
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1107/ ) MAPREDUCE-3927 . Shuffle hang when set map.failures.percent (Bhallamudi Venkata Siva Kamesh via tgraves) (Revision 1348846) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348846 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestShuffleScheduler.java

          People

          • Assignee:
            Bhallamudi Venkata Siva Kamesh
            Reporter:
            MengWang
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development