Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6647

MR usage counters use the resources requested instead of the resources allocated

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As can be seen in the following snippet, the MR counters for usage use the resources requested instead of the resources allocated. The scheduler increment-allocation-mb configs could lead to these values not being the same. We could change the counters to use the allocated resources in order to account for this.

        private static void updateMillisCounters(JobCounterUpdateEvent jce,
            TaskAttemptImpl taskAttempt) {
           /***omitted**/
          long duration = (taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
          int mbRequired =
              taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
          int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, taskType);
      
          int minSlotMemSize = taskAttempt.conf.getInt(
            YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
            YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
      
          int simSlotsRequired =
              minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
                  / minSlotMemSize);
      
          if (taskType == TaskType.MAP) {
            jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * duration);
            jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
            jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * vcoresRequired);
            jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
          } else {
            jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired * duration);
            jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * mbRequired);
            jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * vcoresRequired);
            jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
          }
      
      1. mapreduce6647.001.patch
        12 kB
        Haibo Chen
      2. mapreduce6647.002.patch
        15 kB
        Haibo Chen
      3. mapreduce6647.003.patch
        15 kB
        Haibo Chen
      4. mapreduce6647.004.patch
        13 kB
        Haibo Chen

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 36s trunk passed
          +1 compile 0m 19s trunk passed with JDK v1.8.0_72
          +1 compile 0m 23s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 43s trunk passed
          +1 javadoc 0m 15s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 19s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_72
          +1 javac 0m 17s the patch passed
          +1 compile 0m 20s the patch passed with JDK v1.7.0_95
          +1 javac 0m 20s the patch passed
          -1 checkstyle 0m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 5 new + 228 unchanged - 6 fixed = 233 total (was 234)
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 51s the patch passed
          +1 javadoc 0m 13s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95
          -1 unit 9m 8s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_72.
          -1 unit 9m 44s hadoop-mapreduce-client-app in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          33m 13s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestJobImpl
            hadoop.mapreduce.v2.app.TestRecovery
          JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.app.TestRecovery



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791055/mapreduce6647.001.patch
          JIRA Issue MAPREDUCE-6647
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 96ad989e16db 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67880cc
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 36s trunk passed +1 compile 0m 19s trunk passed with JDK v1.8.0_72 +1 compile 0m 23s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 43s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 19s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_72 +1 javac 0m 17s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed -1 checkstyle 0m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 5 new + 228 unchanged - 6 fixed = 233 total (was 234) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95 -1 unit 9m 8s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_72. -1 unit 9m 44s hadoop-mapreduce-client-app in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 33m 13s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestJobImpl   hadoop.mapreduce.v2.app.TestRecovery JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.app.TestRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791055/mapreduce6647.001.patch JIRA Issue MAPREDUCE-6647 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 96ad989e16db 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67880cc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6358/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          haibochen Haibo Chen added a comment -

          updated patch to take care of test failures in TestRecovery

          Show
          haibochen Haibo Chen added a comment - updated patch to take care of test failures in TestRecovery
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 7m 0s trunk passed
          +1 compile 0m 19s trunk passed with JDK v1.8.0_74
          +1 compile 0m 22s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 42s trunk passed
          +1 javadoc 0m 15s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.8.0_74
          +1 javac 0m 16s the patch passed
          +1 compile 0m 20s the patch passed with JDK v1.7.0_95
          +1 javac 0m 20s the patch passed
          -1 checkstyle 0m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 5 new + 228 unchanged - 6 fixed = 233 total (was 234)
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 53s the patch passed
          +1 javadoc 0m 12s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
          +1 unit 8m 59s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_74.
          +1 unit 9m 37s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          33m 20s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791360/mapreduce6647.002.patch
          JIRA Issue MAPREDUCE-6647
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 872e058693ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0a9f00a
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6360/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6360/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6360/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 7m 0s trunk passed +1 compile 0m 19s trunk passed with JDK v1.8.0_74 +1 compile 0m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 42s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_74 +1 javac 0m 16s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed -1 checkstyle 0m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 5 new + 228 unchanged - 6 fixed = 233 total (was 234) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 8m 59s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_74. +1 unit 9m 37s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 33m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791360/mapreduce6647.002.patch JIRA Issue MAPREDUCE-6647 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 872e058693ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a9f00a Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6360/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6360/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6360/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          haibochen Haibo Chen added a comment -

          updated to take care of checkstyle issue

          Show
          haibochen Haibo Chen added a comment - updated to take care of checkstyle issue
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 6m 44s trunk passed
          +1 compile 0m 19s trunk passed with JDK v1.8.0_74
          +1 compile 0m 22s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 42s trunk passed
          +1 javadoc 0m 15s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_74
          +1 javac 0m 17s the patch passed
          +1 compile 0m 20s the patch passed with JDK v1.7.0_95
          +1 javac 0m 20s the patch passed
          +1 checkstyle 0m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 0 new + 228 unchanged - 6 fixed = 228 total (was 234)
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 52s the patch passed
          +1 javadoc 0m 13s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95
          +1 unit 9m 39s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_74.
          +1 unit 9m 41s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          33m 32s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791371/mapreduce6647.003.patch
          JIRA Issue MAPREDUCE-6647
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9a4493484bfb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ff0ee84
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6361/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6361/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 0m 19s trunk passed with JDK v1.8.0_74 +1 compile 0m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 42s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_74 +1 javac 0m 17s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed +1 checkstyle 0m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 0 new + 228 unchanged - 6 fixed = 228 total (was 234) +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 52s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95 +1 unit 9m 39s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_74. +1 unit 9m 41s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 33m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791371/mapreduce6647.003.patch JIRA Issue MAPREDUCE-6647 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9a4493484bfb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ff0ee84 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6361/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6361/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          Overall looks good. Two minor things:

          • There's some extra newlines in the middle of TestTaskAttempt we don't need
          • I don't think we need MRAppWithConfigurableContainerResource. The parent class, MRApp is already a mock of the real thing, MRAppMaster; so you should be able to change MRApp's code directly. You could add a second constructor that takes a Resource. Using the first constructor would do the original behavior of using Resource.newInstance(1234, 2). We then also don't have to duplicate the handle method's code.
          Show
          rkanter Robert Kanter added a comment - Overall looks good. Two minor things: There's some extra newlines in the middle of TestTaskAttempt we don't need I don't think we need MRAppWithConfigurableContainerResource . The parent class, MRApp is already a mock of the real thing, MRAppMaster ; so you should be able to change MRApp 's code directly. You could add a second constructor that takes a Resource . Using the first constructor would do the original behavior of using Resource.newInstance(1234, 2) . We then also don't have to duplicate the handle method's code.
          Hide
          haibochen Haibo Chen added a comment -

          updated per Robert's comments.

          Show
          haibochen Haibo Chen added a comment - updated per Robert's comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 47s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 6m 56s trunk passed
          +1 compile 0m 19s trunk passed with JDK v1.8.0_77
          +1 compile 0m 22s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 43s trunk passed
          +1 javadoc 0m 14s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.8.0_77
          +1 javac 0m 16s the patch passed
          +1 compile 0m 19s the patch passed with JDK v1.7.0_95
          +1 javac 0m 19s the patch passed
          +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 0 new + 228 unchanged - 6 fixed = 228 total (was 234)
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 53s the patch passed
          +1 javadoc 0m 13s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
          +1 unit 9m 6s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_77.
          +1 unit 9m 41s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 18s Patch generated 1 ASF License warnings.
          44m 46s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795664/mapreduce6647.004.patch
          JIRA Issue MAPREDUCE-6647
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c2f9cf2cc216 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 524bc3c
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6405/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6405/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6405/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 47s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 6m 56s trunk passed +1 compile 0m 19s trunk passed with JDK v1.8.0_77 +1 compile 0m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 43s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 22s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_77 +1 javac 0m 16s the patch passed +1 compile 0m 19s the patch passed with JDK v1.7.0_95 +1 javac 0m 19s the patch passed +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: patch generated 0 new + 228 unchanged - 6 fixed = 228 total (was 234) +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 9m 6s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_77. +1 unit 9m 41s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 18s Patch generated 1 ASF License warnings. 44m 46s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795664/mapreduce6647.004.patch JIRA Issue MAPREDUCE-6647 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c2f9cf2cc216 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 524bc3c Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6405/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6405/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6405/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          +1 Will commit shortly

          Show
          rkanter Robert Kanter added a comment - +1 Will commit shortly
          Hide
          rkanter Robert Kanter added a comment -

          Thanks Haibo. Committed to trunk!

          Show
          rkanter Robert Kanter added a comment - Thanks Haibo. Committed to trunk!
          Hide
          rkanter Robert Kanter added a comment -

          And branch-2!

          Show
          rkanter Robert Kanter added a comment - And branch-2!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9572 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9572/)
          MAPREDUCE-6647. MR usage counters use the resources requested instead of (rkanter: rev 3be1ab485f557c8a3c6a5066453f24d8d61f30be)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9572 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9572/ ) MAPREDUCE-6647 . MR usage counters use the resources requested instead of (rkanter: rev 3be1ab485f557c8a3c6a5066453f24d8d61f30be) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
          Hide
          vrushalic Vrushali C added a comment -

          Hi Haibo Chen Robert Kanter

          Although it's technically a correct use case that we need a counter that reports what the actual usage is, this update will break the meaning for downstream apps which have been using these counter to track their usage. They could now suddenly see a "drop" in their usage of memory or vcores or both.

          Also, how do you envision tracking the resource allocation for these jobs? Is there another counter that tells us how much memory was reserved for this job on the cluster (which is what mbmillis meant before this patch)? Similarly for vcore-millis? Because that memory was not available for scheduling other containers.

          Show
          vrushalic Vrushali C added a comment - Hi Haibo Chen Robert Kanter Although it's technically a correct use case that we need a counter that reports what the actual usage is, this update will break the meaning for downstream apps which have been using these counter to track their usage. They could now suddenly see a "drop" in their usage of memory or vcores or both. Also, how do you envision tracking the resource allocation for these jobs? Is there another counter that tells us how much memory was reserved for this job on the cluster (which is what mbmillis meant before this patch)? Similarly for vcore-millis? Because that memory was not available for scheduling other containers.
          Hide
          vrushalic Vrushali C added a comment -

          Also, this feels like a backwards incompatible for branch-2. Wondering if this should be marked accordingly? It would have been good if we had put in new counters that reflected actual usage and kept the old counters as they are. This patch will now break the meaning of these counter values for all downstream resource tracking systems.

          Show
          vrushalic Vrushali C added a comment - Also, this feels like a backwards incompatible for branch-2. Wondering if this should be marked accordingly? It would have been good if we had put in new counters that reflected actual usage and kept the old counters as they are. This patch will now break the meaning of these counter values for all downstream resource tracking systems.
          Hide
          haibochen Haibo Chen added a comment -

          Thanks for pointing this out Vrushali C! I agree this will change what downstream apps see. But the change is not modifying the counter to report what the actual usage is.
          What it does is that it changes resource requested by tasks to actual resource allocation by YARN, i.e. how much resource was actually reserved for the job. The bug we
          were trying to fix is that if a map task requires 1GB to run, and somehow the minimum allocation of the cluster is set to 2GB, the cluster will actually reserve 2GB for the task.

          Can you please elaborate a little more on how downstream apps make use of these counters?

          Anyway, I think we should revert this from branch-2. Robert Kanter Thoughts?

          Show
          haibochen Haibo Chen added a comment - Thanks for pointing this out Vrushali C ! I agree this will change what downstream apps see. But the change is not modifying the counter to report what the actual usage is. What it does is that it changes resource requested by tasks to actual resource allocation by YARN, i.e. how much resource was actually reserved for the job. The bug we were trying to fix is that if a map task requires 1GB to run, and somehow the minimum allocation of the cluster is set to 2GB, the cluster will actually reserve 2GB for the task. Can you please elaborate a little more on how downstream apps make use of these counters? Anyway, I think we should revert this from branch-2. Robert Kanter Thoughts?
          Hide
          vrushalic Vrushali C added a comment -

          I see, thanks Haibo Chen.

          So with downstream applications like hraven which are used for chargeback on clusters, the accounting for a job is done on the basis of the memory (or vcores) reserved for that job.

          Could we wait till I think about this a bit more before reverting from 2.9?

          As in your example, my thinking is, if YARN reserves 2GB for a container, then 2GB should be accounted towards that job. If YARN reserves 1Gb, then 1 GB goes towards that job. If this is still true after your patch is applied, then we can keep it. We need to account for whatever memory is actually reserved for a job, regardless of what the job actually uses, since that reserved memory is no longer available for other jobs in that queue/cluster.

          Show
          vrushalic Vrushali C added a comment - I see, thanks Haibo Chen . So with downstream applications like hraven which are used for chargeback on clusters, the accounting for a job is done on the basis of the memory (or vcores) reserved for that job. Could we wait till I think about this a bit more before reverting from 2.9? As in your example, my thinking is, if YARN reserves 2GB for a container, then 2GB should be accounted towards that job. If YARN reserves 1Gb, then 1 GB goes towards that job. If this is still true after your patch is applied, then we can keep it. We need to account for whatever memory is actually reserved for a job, regardless of what the job actually uses, since that reserved memory is no longer available for other jobs in that queue/cluster.
          Hide
          rkanter Robert Kanter added a comment -

          I think Haibo Chen's example supports your use case then. If the minimum allocation is 2GB but the user requests 1GB, Yarn will give them 2GB but prior to MR-6647, they'd only be charged for 1GB, which is incorrect.

          Show
          rkanter Robert Kanter added a comment - I think Haibo Chen 's example supports your use case then. If the minimum allocation is 2GB but the user requests 1GB, Yarn will give them 2GB but prior to MR-6647, they'd only be charged for 1GB, which is incorrect.
          Hide
          vrushalic Vrushali C added a comment -

          Got it now. Thanks for the discussion. Am good with the patch.

          Show
          vrushalic Vrushali C added a comment - Got it now. Thanks for the discussion. Am good with the patch.

            People

            • Assignee:
              haibochen Haibo Chen
              Reporter:
              haibochen Haibo Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development