Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      Inside Fetcher.java there are a few cases where an error can happen and the corresponding map task is not marked as a fetch failure. One of these is if the Shuffle server returns a malformed result.

      MAPREDUCE-3992 makes this case a lot less common, but it is still possible. IF the shuffle handler always returns a malformed result, but a OK response the Fetcher will never stop trying to fetch those results.

      1. MR-4423-trunk.txt
        12 kB
        Robert Joseph Evans
      2. MR-4423-trunk.txt
        15 kB
        Robert Joseph Evans
      3. MR-4423-branch-0.23.txt
        15 kB
        Robert Joseph Evans
      4. MR-4423.txt
        11 kB
        Robert Joseph Evans
      5. MR-4423.txt
        11 kB
        Robert Joseph Evans
      6. MR-4423.txt
        12 kB
        Robert Joseph Evans

        Activity

        Hide
        Robert Joseph Evans added a comment -

        This patch adds in a test, and marks tasks failed on any error while trying to fetch the data.

        Show
        Robert Joseph Evans added a comment - This patch adds in a test, and marks tasks failed on any error while trying to fetch the data.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536897/MR-4423.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2616//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2616//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536897/MR-4423.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2616//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2616//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        Looks good overall. One small nit in that the good boolean was removed, but a comment remains that refers to it.

        Show
        Jason Lowe added a comment - Looks good overall. One small nit in that the good boolean was removed, but a comment remains that refers to it.
        Hide
        Robert Joseph Evans added a comment -

        This patch updates the comment that mentioned the deleted good variable.

        Show
        Robert Joseph Evans added a comment - This patch updates the comment that mentioned the deleted good variable.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12537218/MR-4423.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2627//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2627//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537218/MR-4423.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2627//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2627//console This message is automatically generated.
        Hide
        Thomas Graves added a comment -

        +1 looks good. Thanks Bobby! I'll commit this shortly.

        Show
        Thomas Graves added a comment - +1 looks good. Thanks Bobby! I'll commit this shortly.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2500 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2500/)
        MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1363454)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363454
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2500 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2500/ ) MAPREDUCE-4423 . Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1363454) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363454 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2565 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2565/)
        MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1363454)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363454
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2565 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2565/ ) MAPREDUCE-4423 . Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1363454) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363454 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2522 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2522/)
        MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1363454)

        Result = FAILURE
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363454
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2522 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2522/ ) MAPREDUCE-4423 . Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1363454) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363454 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #319 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/319/)
        merge -r 1363454:1363455 from branch-2. FIXES: MAPREDUCE-4423 (Revision 1363456)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363456
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #319 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/319/ ) merge -r 1363454:1363455 from branch-2. FIXES: MAPREDUCE-4423 (Revision 1363456) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363456 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1142/)
        MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1363454)

        Result = FAILURE
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363454
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1142/ ) MAPREDUCE-4423 . Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1363454) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363454 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Thomas Graves added a comment -

        I'm going to revert this because this seems to have uncovered another bug where map output comes back as OK but has internal error in the body. We need to investigate that more.

        Show
        Thomas Graves added a comment - I'm going to revert this because this seems to have uncovered another bug where map output comes back as OK but has internal error in the body. We need to investigate that more.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2573 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2573/)
        svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2573 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2573/ ) svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2508/)
        svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2508/ ) svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2529 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2529/)
        svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935)

        Result = FAILURE
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2529 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2529/ ) svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #320 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/320/)
        svn merge --change -1363456 for reverting MAPREDUCE-4423 (Revision 1363937)

        Result = UNSTABLE
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363937
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #320 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/320/ ) svn merge --change -1363456 for reverting MAPREDUCE-4423 (Revision 1363937) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363937 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1111/)
        svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935)

        Result = FAILURE
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1111/ ) svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1143 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1143/)
        svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935)

        Result = FAILURE
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1143 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1143/ ) svn merge --change -1363454 for reverting MAPREDUCE-4423 (Revision 1363935) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363935 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Robert Joseph Evans added a comment -

        I determined what the problem was. It was not that this patch is flawed, but it was exposing another transient issue that was committed recently MAPREDUCE-4384.

        I did find a few minor issue with the previous patch that I have fixed in this one.

        Show
        Robert Joseph Evans added a comment - I determined what the problem was. It was not that this patch is flawed, but it was exposing another transient issue that was committed recently MAPREDUCE-4384 . I did find a few minor issue with the previous patch that I have fixed in this one.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12537898/MR-4423.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2660//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2660//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537898/MR-4423.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2660//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2660//console This message is automatically generated.
        Hide
        Thomas Graves added a comment -

        doesn't apply cleanly to trunk anymore, can you upmerge.

        Show
        Thomas Graves added a comment - doesn't apply cleanly to trunk anymore, can you upmerge.
        Hide
        Robert Joseph Evans added a comment -

        This version has been upmerged to the latest version of trunk. Please use the original patch for branch-0.23.

        Show
        Robert Joseph Evans added a comment - This version has been upmerged to the latest version of trunk. Please use the original patch for branch-0.23.
        Hide
        Thomas Graves added a comment -

        Still looking, but minor comment, it might be a bit confusing to still log this when you return the empty array when told to wait. I realize the array is empty but still says failed task when it didn't really fail any. Perhaps add a check for empty too.

        if(failedTasks != null) {
        LOG.warn("copyMapOutput failed for tasks "+Arrays.toString(failedTasks));

        Show
        Thomas Graves added a comment - Still looking, but minor comment, it might be a bit confusing to still log this when you return the empty array when told to wait. I realize the array is empty but still says failed task when it didn't really fail any. Perhaps add a check for empty too. if(failedTasks != null) { LOG.warn("copyMapOutput failed for tasks "+Arrays.toString(failedTasks));
        Hide
        Robert Joseph Evans added a comment -

        That is a great catch. I'll update the patches. If you have anything else please let me know.

        Show
        Robert Joseph Evans added a comment - That is a great catch. I'll update the patches. If you have anything else please let me know.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12538008/MR-4423-trunk.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2667//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2667//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538008/MR-4423-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2667//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2667//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        Updated the logging to not warn on WAIT, and added in a test specifically for WAIT.

        Show
        Robert Joseph Evans added a comment - Updated the logging to not warn on WAIT, and added in a test specifically for WAIT.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12538083/MR-4423-trunk.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2668//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2668//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538083/MR-4423-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2668//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2668//console This message is automatically generated.
        Hide
        Thomas Graves added a comment -

        +1 thanks Bobby!

        Show
        Thomas Graves added a comment - +1 thanks Bobby!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2526 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2526/)
        MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1366258)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366258
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2526 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2526/ ) MAPREDUCE-4423 . Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1366258) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366258 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2590 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2590/)
        MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1366258)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366258
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2590 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2590/ ) MAPREDUCE-4423 . Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1366258) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366258 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2546 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2546/)
        MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1366258)

        Result = FAILURE
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366258
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2546 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2546/ ) MAPREDUCE-4423 . Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1366258) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366258 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #326 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/326/)
        MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1366263)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366263
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #326 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/326/ ) MAPREDUCE-4423 . Potential infinite fetching of map output (Robert Evans via tgraves) (Revision 1366263) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366263 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Robert Joseph Evans
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development