Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: mrv2, task
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When running a terasort on latest trunk, I see the following in my task logs:

      2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
      2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal progress value found, progress is larger than 1. Progress will be changed to 1
      2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal progress value found, progress is larger than 1. Progress will be changed to 1
      2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output
      

      We should eliminate these warnings.

      1. MAPREDUCE-6068.002.patch
        1 kB
        Binglin Chang
      2. MAPREDUCE-6068.003.patch
        2 kB
        Binglin Chang
      3. MAPREDUCE-6068.v1.patch
        1 kB
        Binglin Chang

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9081 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9081/)
        MAPREDUCE-6068. Illegal progress value warnings in map tasks. (junping_du: rev 8c1adeaa26a7eaaca891dccdb71ff085e598a7de)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9081 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9081/ ) MAPREDUCE-6068 . Illegal progress value warnings in map tasks. (junping_du: rev 8c1adeaa26a7eaaca891dccdb71ff085e598a7de) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        djp Junping Du added a comment -

        I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Binglin Chang for the contribution and Eduard Camaj for review!

        Show
        djp Junping Du added a comment - I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Binglin Chang for the contribution and Eduard Camaj for review!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 9m 38s trunk passed
        +1 compile 0m 27s trunk passed with JDK v1.8.0_66
        +1 compile 0m 27s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 34s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        -1 findbugs 1m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
        +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 26s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.8.0_66
        +1 javac 0m 20s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_91
        +1 javac 0m 23s the patch passed
        -1 checkstyle 0m 14s Patch generated 1 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core (total was 71, now 72).
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 21s the patch passed
        +1 javadoc 0m 20s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 24s the patch passed with JDK v1.7.0_91
        +1 unit 2m 2s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
        +1 unit 2m 19s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        23m 41s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781488/MAPREDUCE-6068.003.patch
        JIRA Issue MAPREDUCE-6068
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 958657bb7e64 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0e76f1f
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6263/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6263/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6263/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6263/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 38s trunk passed +1 compile 0m 27s trunk passed with JDK v1.8.0_66 +1 compile 0m 27s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 1m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 26s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_66 +1 javac 0m 20s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_91 +1 javac 0m 23s the patch passed -1 checkstyle 0m 14s Patch generated 1 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core (total was 71, now 72). +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 21s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_91 +1 unit 2m 2s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 2m 19s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 23m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781488/MAPREDUCE-6068.003.patch JIRA Issue MAPREDUCE-6068 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 958657bb7e64 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0e76f1f Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6263/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6263/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6263/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6263/console This message was automatically generated.
        Hide
        decster Binglin Chang added a comment -

        Thanks for the review and nice catch Junping Du ! Update the patch.

        Show
        decster Binglin Chang added a comment - Thanks for the review and nice catch Junping Du ! Update the patch.
        Hide
        djp Junping Du added a comment -

        Forget to mention, there is still another place in Merger.java that could over the limit:

                  if (totalBytes != 0)
                    mergeProgress.set(totalBytesProcessed * progPerByte);
                  else
                    mergeProgress.set(1.0f); // Last pass and no segments left - we're done
        

        Binglin Chang, do you think we should update here also?

        Show
        djp Junping Du added a comment - Forget to mention, there is still another place in Merger.java that could over the limit: if (totalBytes != 0) mergeProgress.set(totalBytesProcessed * progPerByte); else mergeProgress.set(1.0f); // Last pass and no segments left - we're done Binglin Chang , do you think we should update here also?
        Hide
        ecamaj Eduard Camaj added a comment -

        Thanks guys.

        Best regards,
        Eddie

        Show
        ecamaj Eduard Camaj added a comment - Thanks guys. Best regards, Eddie
        Hide
        djp Junping Du added a comment -

        002 patch LGTM. +1. Will commit it shortly.

        Show
        djp Junping Du added a comment - 002 patch LGTM. +1. Will commit it shortly.
        Hide
        decster Binglin Chang added a comment -

        Looked findbug warnings and test errors, all unrelated to this code.
        Junping Du Could you help review this?

        Show
        decster Binglin Chang added a comment - Looked findbug warnings and test errors, all unrelated to this code. Junping Du Could you help review this?
        Hide
        ecamaj Eduard Camaj added a comment -

        What about these fails above? And who can decide if this is OK or not?
        It is shame to have fix for such long and people are still facing this error in their deployments with latest version.

        Show
        ecamaj Eduard Camaj added a comment - What about these fails above? And who can decide if this is OK or not? It is shame to have fix for such long and people are still facing this error in their deployments with latest version.
        Hide
        decster Binglin Chang added a comment -

        It's a very simple one line fix, not sure a regression test is needed.

        Show
        decster Binglin Chang added a comment - It's a very simple one line fix, not sure a regression test is needed.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 45s trunk passed
        +1 compile 0m 22s trunk passed with JDK v1.8.0_66
        +1 compile 0m 25s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 32s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        -1 findbugs 1m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
        +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 25s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 18s the patch passed with JDK v1.8.0_66
        +1 javac 0m 18s the patch passed
        +1 compile 0m 22s the patch passed with JDK v1.7.0_91
        +1 javac 0m 22s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 29s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 16s the patch passed
        +1 javadoc 0m 19s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91
        +1 unit 1m 52s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
        -1 unit 2m 11s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        20m 53s



        Reason Tests
        JDK v1.7.0_91 Failed junit tests hadoop.mapreduce.tools.TestCLI



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12687161/MAPREDUCE-6068.002.patch
        JIRA Issue MAPREDUCE-6068
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8693cbabe381 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 96d8f1d
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 45s trunk passed +1 compile 0m 22s trunk passed with JDK v1.8.0_66 +1 compile 0m 25s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 13s trunk passed -1 findbugs 1m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 24s the patch passed +1 compile 0m 18s the patch passed with JDK v1.8.0_66 +1 javac 0m 18s the patch passed +1 compile 0m 22s the patch passed with JDK v1.7.0_91 +1 javac 0m 22s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91 +1 unit 1m 52s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. -1 unit 2m 11s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 20m 53s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.mapreduce.tools.TestCLI Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12687161/MAPREDUCE-6068.002.patch JIRA Issue MAPREDUCE-6068 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8693cbabe381 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 96d8f1d Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6254/console This message was automatically generated.
        Hide
        ecamaj Eduard Camaj added a comment -

        We also encountered this bug.
        Still not merged - because of missing test?
        Binglin Chang, could you fix this please - add or elaborate (I don't know to who and where), as stated above?
        thanks

        Show
        ecamaj Eduard Camaj added a comment - We also encountered this bug. Still not merged - because of missing test? Binglin Chang , could you fix this please - add or elaborate (I don't know to who and where), as stated above? thanks
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12687161/MAPREDUCE-6068.002.patch
        against trunk revision af9d4fe.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5384//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5384//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12687161/MAPREDUCE-6068.002.patch against trunk revision af9d4fe. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5384//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5384//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12687161/MAPREDUCE-6068.002.patch
        against trunk revision 25a0440.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5080//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5080//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5080//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12687161/MAPREDUCE-6068.002.patch against trunk revision 25a0440. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5080//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5080//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5080//console This message is automatically generated.
        Hide
        decster Binglin Chang added a comment -

        Rebase to trunk

        Show
        decster Binglin Chang added a comment - Rebase to trunk
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12668020/MAPREDUCE-6068.v1.patch
        against trunk revision 4be9517.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4867//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4867//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668020/MAPREDUCE-6068.v1.patch against trunk revision 4be9517. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4867//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4867//console This message is automatically generated.
        Hide
        decster Binglin Chang added a comment -

        Add some trace log:

        2014-09-11 17:49:38,282 WARN  [pool-65-thread-1] mapred.Merger (Merger.java:next(566)) - totalBytesProcessed: 12076353 progPerByte: 8.294227E-8 result: 1.0016402
        2014-09-11 17:49:38,282 WARN  [pool-65-thread-1] util.Progress (Progress.java:set(180)) - Illegal progress value found, progress is larger than 1. Progress will be changed to 1
        

        Merger use estimated progress calculation, when data size is large, it is possible that the estimated value is a little above 1.0.
        The patch makes sure Merger will never set progress larger than 1.0.

        More thoughts(not related to this jira) on this part of code, the current code set progress 1 or 2 times for each k/v pairs, Progress.set is synchronized, and the calculation and code in Progress.set involve a lot floating point operations, this part of code may impact performance a little, though I have not tested.

        Show
        decster Binglin Chang added a comment - Add some trace log: 2014-09-11 17:49:38,282 WARN [pool-65-thread-1] mapred.Merger (Merger.java:next(566)) - totalBytesProcessed: 12076353 progPerByte: 8.294227E-8 result: 1.0016402 2014-09-11 17:49:38,282 WARN [pool-65-thread-1] util.Progress (Progress.java:set(180)) - Illegal progress value found, progress is larger than 1. Progress will be changed to 1 Merger use estimated progress calculation, when data size is large, it is possible that the estimated value is a little above 1.0. The patch makes sure Merger will never set progress larger than 1.0. More thoughts(not related to this jira) on this part of code, the current code set progress 1 or 2 times for each k/v pairs, Progress.set is synchronized, and the calculation and code in Progress.set involve a lot floating point operations, this part of code may impact performance a little, though I have not tested.

          People

          • Assignee:
            decster Binglin Chang
            Reporter:
            tlipcon Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development