Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6605

Fix typos mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc.count.autoincr in mapred-default.xml

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As the default configuration file shows,
      https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

      we can configure options
      mapreduce.map.skip.proc.count.autoincr
      mapreduce.reduce.skip.proc.count.autoincr
      in the mapred-default.xml. But they do not work because the expected keys in org.apache.hadoop.mapreduce.MRJobConfig.java

      public static final String MAP_SKIP_INCR_PROC_COUNT = "mapreduce.map.skip.proc-count.auto-incr"
      public static final String REDUCE_SKIP_INCR_PROC_COUNT = "mapreduce.reduce.skip.proc-count.auto-incr";

      Or
      in org.apache.hadoop.mapreduce.util.ConfigUtil.java

      new DeprecationDelta("mapred.skip.map.auto.incr.proc.count",
      MRJobConfig.MAP_SKIP_INCR_PROC_COUNT),
      new DeprecationDelta("mapred.skip.reduce.auto.incr.proc.count",
      MRJobConfig.REDUCE_SKIP_INCR_PROC_COUNT),

      we can change them to mapreduce.map.skip.proc-count.auto-incr and mapreduce.reduce.skip.proc-count.auto-incr in the default configuration file.

      1. MAPREDUCE-6605.01.patch
        2 kB
        Kai Sasaki
      2. MAPREDUCE-6605-branch-2.01.patch
        7 kB
        Kai Sasaki

        Issue Links

          Activity

          Hide
          lewuathe Kai Sasaki added a comment -

          In the current HEAD, these configurations are regarded as obsolete and removed in MAPREDUCE-6057.
          https://issues.apache.org/jira/browse/MAPREDUCE-6057
          There is no entry about SkipBadRecords#COUNTER_[MAP/REDUCE]_PROCESSED_RECORDS in mapred-default.xml. My patch added the entries with the default values.

          Show
          lewuathe Kai Sasaki added a comment - In the current HEAD, these configurations are regarded as obsolete and removed in MAPREDUCE-6057 . https://issues.apache.org/jira/browse/MAPREDUCE-6057 There is no entry about SkipBadRecords#COUNTER_[MAP/REDUCE]_PROCESSED_RECORDS in mapred-default.xml. My patch added the entries with the default values.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 40s trunk passed
          +1 compile 0m 21s trunk passed with JDK v1.8.0_66
          +1 compile 0m 24s trunk passed with JDK v1.7.0_91
          +1 mvnsite 0m 32s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 25s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 25s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_66
          +1 javac 0m 17s the patch passed
          +1 compile 0m 22s the patch passed with JDK v1.7.0_91
          +1 javac 0m 22s the patch passed
          +1 mvnsite 0m 29s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 18s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 23s the patch passed with JDK v1.7.0_91
          -1 unit 1m 49s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_66.
          +1 unit 2m 13s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          17m 45s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.tools.TestCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782766/MAPREDUCE-6605.01.patch
          JIRA Issue MAPREDUCE-6605
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 21cc27cea6b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / da77f42
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6273/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6273/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6273/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6273/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 40s trunk passed +1 compile 0m 21s trunk passed with JDK v1.8.0_66 +1 compile 0m 24s trunk passed with JDK v1.7.0_91 +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 14s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 25s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_66 +1 javac 0m 17s the patch passed +1 compile 0m 22s the patch passed with JDK v1.7.0_91 +1 javac 0m 22s the patch passed +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 18s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_91 -1 unit 1m 49s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_66. +1 unit 2m 13s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 17m 45s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.tools.TestCLI Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782766/MAPREDUCE-6605.01.patch JIRA Issue MAPREDUCE-6605 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 21cc27cea6b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / da77f42 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6273/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6273/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6273/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6273/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          The patch looks good to me. Hi Kai Sasaki, would you create another patch to fix the typo for branch-2? The typo is not only in mapred-default.xml but also in TestMapreduceConfigFields.java and some other files.

          Show
          ajisakaa Akira Ajisaka added a comment - The patch looks good to me. Hi Kai Sasaki , would you create another patch to fix the typo for branch-2? The typo is not only in mapred-default.xml but also in TestMapreduceConfigFields.java and some other files.
          Hide
          lewuathe Kai Sasaki added a comment -

          Akira Ajisaka Sure. I'll do that. Thanks for pointing out.

          Show
          lewuathe Kai Sasaki added a comment - Akira Ajisaka Sure. I'll do that. Thanks for pointing out.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 2m 21s Maven dependency ordering for branch
          +1 mvninstall 7m 23s branch-2 passed
          +1 compile 5m 53s branch-2 passed with JDK v1.8.0_66
          +1 compile 6m 37s branch-2 passed with JDK v1.7.0_91
          +1 checkstyle 0m 59s branch-2 passed
          +1 mvnsite 1m 50s branch-2 passed
          +1 mvneclipse 0m 59s branch-2 passed
          -1 findbugs 1m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in branch-2 has 2 extant Findbugs warnings.
          +1 javadoc 1m 9s branch-2 passed with JDK v1.8.0_66
          +1 javadoc 1m 16s branch-2 passed with JDK v1.7.0_91
          0 mvndep 0m 35s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 6m 18s the patch passed with JDK v1.8.0_66
          +1 javac 6m 18s the patch passed
          +1 compile 6m 34s the patch passed with JDK v1.7.0_91
          +1 javac 6m 34s the patch passed
          +1 checkstyle 0m 57s the patch passed
          +1 mvnsite 1m 48s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 58s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 17s the patch passed with JDK v1.7.0_91
          +1 unit 1m 54s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 56s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_66.
          +1 unit 5m 41s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 53s hadoop-sls in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 12s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 37s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91.
          +1 unit 6m 1s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_91.
          +1 unit 0m 57s hadoop-sls in the patch passed with JDK v1.7.0_91.
          -1 asflicense 0m 22s Patch generated 14 ASF License warnings.
          94m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783778/MAPREDUCE-6605-branch-2.01.patch
          JIRA Issue MAPREDUCE-6605
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 40d7eae3d300 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 73ef77c
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6274/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6274/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6274/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-sls U: .
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6274/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 21s Maven dependency ordering for branch +1 mvninstall 7m 23s branch-2 passed +1 compile 5m 53s branch-2 passed with JDK v1.8.0_66 +1 compile 6m 37s branch-2 passed with JDK v1.7.0_91 +1 checkstyle 0m 59s branch-2 passed +1 mvnsite 1m 50s branch-2 passed +1 mvneclipse 0m 59s branch-2 passed -1 findbugs 1m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in branch-2 has 2 extant Findbugs warnings. +1 javadoc 1m 9s branch-2 passed with JDK v1.8.0_66 +1 javadoc 1m 16s branch-2 passed with JDK v1.7.0_91 0 mvndep 0m 35s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 6m 18s the patch passed with JDK v1.8.0_66 +1 javac 6m 18s the patch passed +1 compile 6m 34s the patch passed with JDK v1.7.0_91 +1 javac 6m 34s the patch passed +1 checkstyle 0m 57s the patch passed +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 55s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 58s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 17s the patch passed with JDK v1.7.0_91 +1 unit 1m 54s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 8m 56s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_66. +1 unit 5m 41s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_66. +1 unit 0m 53s hadoop-sls in the patch passed with JDK v1.8.0_66. +1 unit 2m 12s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 unit 9m 37s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_91. +1 unit 6m 1s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_91. +1 unit 0m 57s hadoop-sls in the patch passed with JDK v1.7.0_91. -1 asflicense 0m 22s Patch generated 14 ASF License warnings. 94m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783778/MAPREDUCE-6605-branch-2.01.patch JIRA Issue MAPREDUCE-6605 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 40d7eae3d300 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 73ef77c Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6274/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6274/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6274/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-tools/hadoop-sls U: . Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6274/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Kai Sasaki for updating the patch. +1, committing this.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Kai Sasaki for updating the patch. +1, committing this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk, branch-2, and branch-2.8. Thanks Kai Sasaki for the contribution!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, and branch-2.8. Thanks Kai Sasaki for the contribution!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9162 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9162/)
          MAPREDUCE-6605. Fix typos mapreduce.map.skip.proc.count.autoincr and (aajisaka: rev f5c8c857beea5b1c0a652f3097c15c06c421fbaa)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9162 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9162/ ) MAPREDUCE-6605 . Fix typos mapreduce.map.skip.proc.count.autoincr and (aajisaka: rev f5c8c857beea5b1c0a652f3097c15c06c421fbaa) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Reviewing the patch, I found the following code in branch-2 can be removed. Filed MAPREDUCE-6614.

          TestMapreduceConfigFields.java
              xmlPropsToSkipCompare.add("mapreduce.reduce.skip.proc-count.auto-incr");
              xmlPropsToSkipCompare.add("mapreduce.map.skip.proc-count.auto-incr");
          
          Show
          ajisakaa Akira Ajisaka added a comment - Reviewing the patch, I found the following code in branch-2 can be removed. Filed MAPREDUCE-6614 . TestMapreduceConfigFields.java xmlPropsToSkipCompare.add( "mapreduce.reduce.skip.proc-count.auto-incr" ); xmlPropsToSkipCompare.add( "mapreduce.map.skip.proc-count.auto-incr" );

            People

            • Assignee:
              lewuathe Kai Sasaki
              Reporter:
              zhendong Dong Zhen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development