Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6795

Update the document for JobConf#setNumReduceTasks

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following document is for MRv1. We should update the document for MapReduce on YARN.

      JobConf.java
         * <b id="NoOfReduces">How many reduces?</b>
         * 
         * <p>The right number of reduces seems to be <code>0.95</code> or 
         * <code>1.75</code> multiplied by (&lt;<i>no. of nodes</i>&gt; * 
         * <a href="{@docRoot}/../mapred-default.html#mapreduce.tasktracker.reduce.tasks.maximum">
         * mapreduce.tasktracker.reduce.tasks.maximum</a>).
         * </p>
      
      1. MAPREDUCE-6795.001.patch
        1 kB
        Yiqun Lin
      2. MAPREDUCE-6795.002.patch
        2 kB
        Yiqun Lin

        Issue Links

          Activity

          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Akira Ajisaka for the patient review and commit!

          Show
          linyiqun Yiqun Lin added a comment - Thanks Akira Ajisaka for the patient review and commit!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10743 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10743/)
          MAPREDUCE-6795. Update the document for JobConf#setNumReduceTasks. (aajisaka: rev 235f39b7c44c5907bd4de063892905577c35cd80)

          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10743 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10743/ ) MAPREDUCE-6795 . Update the document for JobConf#setNumReduceTasks. (aajisaka: rev 235f39b7c44c5907bd4de063892905577c35cd80) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk, branch-2, and branch-2.8. Thanks Yiqun Lin for the contribution!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, and branch-2.8. Thanks Yiqun Lin for the contribution!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1, committing this.

          Show
          ajisakaa Akira Ajisaka added a comment - +1, committing this.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 54s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 38s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 7s trunk passed
          +1 javadoc 0m 25s trunk passed
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          -1 checkstyle 0m 20s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 1 new + 90 unchanged - 1 fixed = 91 total (was 91)
          +1 mvnsite 0m 33s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 54s the patch passed
          +1 javadoc 0m 18s the patch passed
          +1 unit 2m 46s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          19m 32s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836137/MAPREDUCE-6795.002.patch
          JIRA Issue MAPREDUCE-6795
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1625eccd3400 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f646fe3
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6790/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6790/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6790/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 54s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 7s trunk passed +1 javadoc 0m 25s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed -1 checkstyle 0m 20s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 1 new + 90 unchanged - 1 fixed = 91 total (was 91) +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 54s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 2m 46s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 19m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836137/MAPREDUCE-6795.002.patch JIRA Issue MAPREDUCE-6795 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1625eccd3400 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f646fe3 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6790/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6790/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6790/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          I am sorry for ignoring that, reupload v002 patch again.

          Show
          linyiqun Yiqun Lin added a comment - I am sorry for ignoring that, reupload v002 patch again.
          Hide
          ajisakaa Akira Ajisaka added a comment -
             * <a href="{@docRoot}/../hadoop-mapreduce-client/hadoop-mapreduce-client-
             * core/mapred-default.xml#mapreduce.reduce.memory.mb">
          

          Would you make this one line to remove unnecessary whitespace from the link? I'm +1 if that is addressed.

          Show
          ajisakaa Akira Ajisaka added a comment - * <a href="{@docRoot}/../hadoop-mapreduce-client/hadoop-mapreduce-client- * core/mapred- default .xml#mapreduce.reduce.memory.mb"> Would you make this one line to remove unnecessary whitespace from the link? I'm +1 if that is addressed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 1s trunk passed
          +1 compile 0m 24s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 30s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 49s trunk passed
          +1 javadoc 0m 22s trunk passed
          +1 mvninstall 0m 25s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          +1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 0 new + 90 unchanged - 1 fixed = 90 total (was 91)
          +1 mvnsite 0m 26s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 55s the patch passed
          -1 javadoc 0m 18s hadoop-mapreduce-client-core in the patch failed.
          +1 unit 2m 46s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          16m 17s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836131/MAPREDUCE-6795.002.patch
          JIRA Issue MAPREDUCE-6795
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 27517cf4470e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f646fe3
          Default Java 1.8.0_101
          findbugs v3.0.0
          javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6789/artifact/patchprocess/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6789/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6789/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 1s trunk passed +1 compile 0m 24s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 25s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 0 new + 90 unchanged - 1 fixed = 90 total (was 91) +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 55s the patch passed -1 javadoc 0m 18s hadoop-mapreduce-client-core in the patch failed. +1 unit 2m 46s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 16m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836131/MAPREDUCE-6795.002.patch JIRA Issue MAPREDUCE-6795 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 27517cf4470e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f646fe3 Default Java 1.8.0_101 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6789/artifact/patchprocess/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6789/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6789/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Akira Ajisaka for the quick review and comments! Post the new patch to address the comment.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Akira Ajisaka for the quick review and comments! Post the new patch to address the comment.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 42s trunk passed
          +1 compile 0m 23s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 47s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          +1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 0 new + 90 unchanged - 1 fixed = 90 total (was 91)
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 51s the patch passed
          +1 javadoc 0m 19s the patch passed
          +1 unit 2m 41s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          15m 48s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836127/MAPREDUCE-6795.001.patch
          JIRA Issue MAPREDUCE-6795
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 67014b80dade 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f646fe3
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6788/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6788/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 42s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 47s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 0 new + 90 unchanged - 1 fixed = 90 total (was 91) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 2m 41s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 15m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836127/MAPREDUCE-6795.001.patch JIRA Issue MAPREDUCE-6795 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 67014b80dade 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f646fe3 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6788/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6788/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Yiqun Lin for updating the patch.

          <a href="{@docRoot}/../mapred-default.html#mapreduce.reduce.memory.mb">
          

          Would you fix this dead link as follows?

          <a href="{@docRoot}/../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml#mapreduce.reduce.memory.mb">
          
          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Yiqun Lin for updating the patch. <a href= "{@docRoot}/../mapred- default .html#mapreduce.reduce.memory.mb" > Would you fix this dead link as follows? <a href= "{@docRoot}/../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred- default .xml#mapreduce.reduce.memory.mb" >
          Hide
          linyiqun Yiqun Lin added a comment -

          Reupload the v001 patch. I think it will be better to explain the available memory for reduce tasks here. Make a minor change based on the original patch.

          Show
          linyiqun Yiqun Lin added a comment - Reupload the v001 patch. I think it will be better to explain the available memory for reduce tasks here. Make a minor change based on the original patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 48s trunk passed
          +1 compile 0m 23s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 29s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 48s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 0 new + 90 unchanged - 1 fixed = 90 total (was 91)
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 52s the patch passed
          +1 javadoc 0m 18s the patch passed
          +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          15m 51s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836117/MAPREDUCE-6795.001.patch
          JIRA Issue MAPREDUCE-6795
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 45a40af43d2d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e0bebbb
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6786/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6786/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 48s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 48s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 0 new + 90 unchanged - 1 fixed = 90 total (was 91) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 52s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 15m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836117/MAPREDUCE-6795.001.patch JIRA Issue MAPREDUCE-6795 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 45a40af43d2d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e0bebbb Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6786/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6786/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Akira Ajisaka for sharing your thought.

          After all, "multiplied by (<available memory for reduce tasks> / mapreduce.reduce.memory.mb)" is good to me.

          LGTM too. Attach a patch for this.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Akira Ajisaka for sharing your thought. After all, "multiplied by (<available memory for reduce tasks> / mapreduce.reduce.memory.mb)" is good to me. LGTM too. Attach a patch for this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I'm thinking it's more difficult to tune the value in MRv2 than MRv1.

          I think the value numNodes * mapreduce.tasktracker.reduce.tasks.maximum in MRV1 is equal to the current config mapreduce.job.running.reduce.limit.

          mapreduce.job.running.reduce.limit is an optional parameter (MAPREDUCE-5583). The number of reduce tasks is limited to "yarn.nodemanager.resource.memory-mb / mapreduce.reduce.memory.mb", but the resource is shared by map tasks and other applications. Therefore the limit of the number of reduce tasks becomes smaller than "yarn.nodemanager.resource.memory-mb / mapreduce.reduce.memory.mb" if some map tasks or other applications are running.

          After all, "multiplied by (<available memory for reduce tasks> / mapreduce.reduce.memory.mb)" is good to me.

          Show
          ajisakaa Akira Ajisaka added a comment - I'm thinking it's more difficult to tune the value in MRv2 than MRv1. I think the value numNodes * mapreduce.tasktracker.reduce.tasks.maximum in MRV1 is equal to the current config mapreduce.job.running.reduce.limit. mapreduce.job.running.reduce.limit is an optional parameter ( MAPREDUCE-5583 ). The number of reduce tasks is limited to "yarn.nodemanager.resource.memory-mb / mapreduce.reduce.memory.mb", but the resource is shared by map tasks and other applications. Therefore the limit of the number of reduce tasks becomes smaller than "yarn.nodemanager.resource.memory-mb / mapreduce.reduce.memory.mb" if some map tasks or other applications are running. After all, "multiplied by (<available memory for reduce tasks> / mapreduce.reduce.memory.mb)" is good to me.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Akira Ajisaka for reporting this. I'd like to make a quick fix. But it seems that I am not a contributor of MAPREDUCE now and I can't upload my patch. The patch file created in my local:

          diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          index f2b0aae..c8b0458 100644
          --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          @@ -1347,9 +1347,9 @@ public void setReduceSpeculativeExecution(boolean speculativeExecution) {
              * <b id="NoOfReduces">How many reduces?</b>
              * 
              * <p>The right number of reduces seems to be <code>0.95</code> or 
          -   * <code>1.75</code> multiplied by (&lt;<i>no. of nodes</i>&gt; * 
          -   * <a href="{@docRoot}/../mapred-default.html#mapreduce.tasktracker.reduce.tasks.maximum">
          -   * mapreduce.tasktracker.reduce.tasks.maximum</a>).
          +   * <code>1.75</code> multiplied by (
          +   * <a href="{@docRoot}/../mapred-default.html#mapreduce.job.running.reduce.limit">
          +   * mapreduce.job.running.reduce.limit</a>).
              * </p>
              * 
              * <p>With <code>0.95</code> all of the reduces can launch immediately and 
          

          I think the value numNodes * mapreduce.tasktracker.reduce.tasks.maximum in MRV1 is equal to the current config mapreduce.job.running.reduce.limit. Correct me if I am wrong. Thanks!

          Show
          linyiqun Yiqun Lin added a comment - Thanks Akira Ajisaka for reporting this. I'd like to make a quick fix. But it seems that I am not a contributor of MAPREDUCE now and I can't upload my patch. The patch file created in my local: diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java index f2b0aae..c8b0458 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java @@ -1347,9 +1347,9 @@ public void setReduceSpeculativeExecution( boolean speculativeExecution) { * <b id= "NoOfReduces" >How many reduces?</b> * * <p>The right number of reduces seems to be <code>0.95</code> or - * <code>1.75</code> multiplied by (&lt;<i>no. of nodes</i>&gt; * - * <a href= "{@docRoot}/../mapred- default .html#mapreduce.tasktracker.reduce.tasks.maximum" > - * mapreduce.tasktracker.reduce.tasks.maximum</a>). + * <code>1.75</code> multiplied by ( + * <a href= "{@docRoot}/../mapred- default .html#mapreduce.job.running.reduce.limit" > + * mapreduce.job.running.reduce.limit</a>). * </p> * * <p>With <code>0.95</code> all of the reduces can launch immediately and I think the value numNodes * mapreduce.tasktracker.reduce.tasks.maximum in MRV1 is equal to the current config mapreduce.job.running.reduce.limit . Correct me if I am wrong. Thanks!

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              ajisakaa Akira Ajisaka
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development