Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which are snapshots of memory usage of that task. They are not sufficient for users to understand peak memory usage by that task, e.g. in order to diagnose task failures, tune job parameters or change application design. This new feature will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and VIRTUAL_MEMORY_BYTES_MAX.

      This JIRA has the same feature from MAPREDUCE-4710. I file this new YARN JIRA since MAPREDUCE-4710 is pretty old one from MR 1.x era, it more or less assumes a branch-1 architecture, should be close at this point.

      1. MAPREDUCE-6829.000.patch
        7 kB
        Miklos Szegedi
      2. MAPREDUCE-6829.001.patch
        14 kB
        Miklos Szegedi
      3. MAPREDUCE-6829.002.patch
        14 kB
        Miklos Szegedi
      4. MAPREDUCE-6829.003.patch
        14 kB
        Miklos Szegedi
      5. MAPREDUCE-6829.004.patch
        14 kB
        Miklos Szegedi
      6. MAPREDUCE-6829.005.patch
        14 kB
        Miklos Szegedi

        Issue Links

          Activity

          Hide
          vvasudev Varun Vasudev added a comment -

          Yufei Gu - is this work for every container? Have you looked at the ContainerMetrics class - it should have this information for physical memory at least.

          Show
          vvasudev Varun Vasudev added a comment - Yufei Gu - is this work for every container? Have you looked at the ContainerMetrics class - it should have this information for physical memory at least.
          Hide
          yufeigu Yufei Gu added a comment -

          Varun Vasudev, thanks for pointing out. But I cannot find these metrics on JHS or RM WebUI when a job is running or after it's done. Seems like it is used only internally.

          Show
          yufeigu Yufei Gu added a comment - Varun Vasudev , thanks for pointing out. But I cannot find these metrics on JHS or RM WebUI when a job is running or after it's done. Seems like it is used only internally.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Attaching a patch that adds two memory peak counters.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Attaching a patch that adds two memory peak counters.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 14m 30s trunk passed
          +1 compile 0m 25s trunk passed
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 47s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 24s the patch passed
          +1 compile 0m 21s the patch passed
          +1 javac 0m 21s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 58s the patch passed
          +1 javadoc 0m 24s the patch passed
          +1 unit 2m 54s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          24m 9s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845889/MAPREDUCE-6829.000.patch
          JIRA Issue MAPREDUCE-6829
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ca4cbe8e8c8a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 02766b6
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6861/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6861/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 30s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 47s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 24s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 58s the patch passed +1 javadoc 0m 24s the patch passed +1 unit 2m 54s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 24m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845889/MAPREDUCE-6829.000.patch JIRA Issue MAPREDUCE-6829 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ca4cbe8e8c8a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 02766b6 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6861/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6861/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          Patch looks pretty good. Few comments:

          1. I see we are operating based on the _MAX suffix. What happens to user counters that end in _MAX? Is there a way to skip them?
          2. We don't need to handle MIN. My recommendation would be to avoid adding it until it is needed. I am okay with keeping them in if you insist, but would like to see a test.
          3. Can we also add a test that uses MiniMRCluster so we have an end-to-end test as well.
          Show
          kasha Karthik Kambatla added a comment - Patch looks pretty good. Few comments: I see we are operating based on the _MAX suffix. What happens to user counters that end in _MAX? Is there a way to skip them? We don't need to handle MIN. My recommendation would be to avoid adding it until it is needed. I am okay with keeping them in if you insist, but would like to see a test. Can we also add a test that uses MiniMRCluster so we have an end-to-end test as well.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Thank you, Karthik Kambatla for the review.
          1. There are two aggregation options. MAX and SUM. If the counter name ends with _MAX we choose the maximum of all container values, we sum them up otherwise.
          2. I removed MIN in the next patch
          3. I added the end-to-end test in the next patch.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you, Karthik Kambatla for the review. 1. There are two aggregation options. MAX and SUM. If the counter name ends with _MAX we choose the maximum of all container values, we sum them up otherwise. 2. I removed MIN in the next patch 3. I added the end-to-end test in the next patch.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Updating patch based on comments

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Updating patch based on comments
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 54s Maven dependency ordering for branch
          +1 mvninstall 14m 49s trunk passed
          +1 compile 1m 45s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 32s trunk passed
          +1 findbugs 1m 27s trunk passed
          +1 javadoc 0m 35s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 52s the patch passed
          +1 compile 2m 1s the patch passed
          +1 javac 2m 1s the patch passed
          -1 checkstyle 0m 32s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 8 new + 287 unchanged - 0 fixed = 295 total (was 287)
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 39s the patch passed
          +1 javadoc 0m 27s the patch passed
          +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 97m 53s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          130m 40s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847978/MAPREDUCE-6829.001.patch
          JIRA Issue MAPREDUCE-6829
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4ed13e6c6f76 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e224c96
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6869/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6869/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6869/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 54s Maven dependency ordering for branch +1 mvninstall 14m 49s trunk passed +1 compile 1m 45s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 32s trunk passed +1 findbugs 1m 27s trunk passed +1 javadoc 0m 35s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 52s the patch passed +1 compile 2m 1s the patch passed +1 javac 2m 1s the patch passed -1 checkstyle 0m 32s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 8 new + 287 unchanged - 0 fixed = 295 total (was 287) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 39s the patch passed +1 javadoc 0m 27s the patch passed +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed. +1 unit 97m 53s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 130m 40s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847978/MAPREDUCE-6829.001.patch JIRA Issue MAPREDUCE-6829 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4ed13e6c6f76 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e224c96 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6869/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6869/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6869/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Fixing checkstyle.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Fixing checkstyle.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 14m 42s trunk passed
          +1 compile 1m 41s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 1m 16s trunk passed
          +1 javadoc 0m 35s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 0m 46s the patch passed
          +1 compile 1m 43s the patch passed
          +1 javac 1m 43s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 29s the patch passed
          +1 javadoc 0m 32s the patch passed
          +1 unit 2m 54s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 115m 42s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          146m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848100/MAPREDUCE-6829.002.patch
          JIRA Issue MAPREDUCE-6829
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 11a9cbcc7f3a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9130af3
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6870/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6870/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 14m 42s trunk passed +1 compile 1m 41s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 1m 16s trunk passed +1 javadoc 0m 35s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 46s the patch passed +1 compile 1m 43s the patch passed +1 javac 1m 43s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 29s the patch passed +1 javadoc 0m 32s the patch passed +1 unit 2m 54s hadoop-mapreduce-client-core in the patch passed. +1 unit 115m 42s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 146m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848100/MAPREDUCE-6829.002.patch JIRA Issue MAPREDUCE-6829 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 11a9cbcc7f3a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9130af3 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6870/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6870/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Unit test change to make sure customer counters are not affected by the change.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Unit test change to make sure customer counters are not affected by the change.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 54s Maven dependency ordering for branch
          +1 mvninstall 13m 38s trunk passed
          +1 compile 1m 51s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 32s trunk passed
          +1 findbugs 1m 24s trunk passed
          +1 javadoc 0m 35s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 43s the patch passed
          +1 compile 1m 31s the patch passed
          +1 javac 1m 31s the patch passed
          -1 checkstyle 0m 29s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 1 new + 287 unchanged - 0 fixed = 288 total (was 287)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 34s the patch passed
          +1 javadoc 0m 29s the patch passed
          +1 unit 2m 51s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 104m 51s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          135m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849196/MAPREDUCE-6829.003.patch
          JIRA Issue MAPREDUCE-6829
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 18e1f12c53f6 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8528d85
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6877/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6877/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6877/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 54s Maven dependency ordering for branch +1 mvninstall 13m 38s trunk passed +1 compile 1m 51s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 32s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 35s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 43s the patch passed +1 compile 1m 31s the patch passed +1 javac 1m 31s the patch passed -1 checkstyle 0m 29s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 1 new + 287 unchanged - 0 fixed = 288 total (was 287) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 34s the patch passed +1 javadoc 0m 29s the patch passed +1 unit 2m 51s hadoop-mapreduce-client-core in the patch passed. +1 unit 104m 51s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 135m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849196/MAPREDUCE-6829.003.patch JIRA Issue MAPREDUCE-6829 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 18e1f12c53f6 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8528d85 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6877/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6877/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6877/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Fixing checkstyle

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Fixing checkstyle
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 14m 40s trunk passed
          +1 compile 2m 2s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          +1 findbugs 1m 29s trunk passed
          +1 javadoc 0m 38s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 2m 0s the patch passed
          +1 javac 2m 0s the patch passed
          +1 checkstyle 0m 32s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 25s the patch passed
          +1 javadoc 0m 32s the patch passed
          +1 unit 2m 58s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 110m 49s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          143m 11s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849327/MAPREDUCE-6829.004.patch
          JIRA Issue MAPREDUCE-6829
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5af5a55a3852 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5a56520
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6878/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6878/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 14m 40s trunk passed +1 compile 2m 2s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 1m 29s trunk passed +1 javadoc 0m 38s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 2m 0s the patch passed +1 javac 2m 0s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 25s the patch passed +1 javadoc 0m 32s the patch passed +1 unit 2m 58s hadoop-mapreduce-client-core in the patch passed. +1 unit 110m 49s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 143m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849327/MAPREDUCE-6829.004.patch JIRA Issue MAPREDUCE-6829 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5af5a55a3852 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5a56520 Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6878/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6878/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          The patch looks good, but for the following nits:

          1. Can we import Mapper and Reducer so the definition of classes TokenizerMapper and IntSumReducer can fit in two lines, may be split the line after extends?
          2. TokenizerMapper#map method indentation looks funny. How about the following:
                public void map(Object key, Text value, Context context) 
                    throws IOException, InterruptedException {
            
          3. IntSumReducer#reduce method indentation looks funny. How about the following:
                public void reduce(Text key, Iterable<IntWritable> values, Context context)
                    throws IOException, InterruptedException {
            
          4. The InternalClass seems unnecessary. If the intent is just to be able to identify the caller jar, why not just use this.getClass()?
          Show
          kasha Karthik Kambatla added a comment - The patch looks good, but for the following nits: Can we import Mapper and Reducer so the definition of classes TokenizerMapper and IntSumReducer can fit in two lines, may be split the line after extends? TokenizerMapper#map method indentation looks funny. How about the following: public void map( Object key, Text value, Context context) throws IOException, InterruptedException { IntSumReducer#reduce method indentation looks funny. How about the following: public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { The InternalClass seems unnecessary. If the intent is just to be able to identify the caller jar, why not just use this.getClass()?
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Thank you for the review Karthik Kambatla. I updated the patch based on your suggestions.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you for the review Karthik Kambatla . I updated the patch based on your suggestions.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 12m 37s trunk passed
          +1 compile 1m 35s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 1m 10s trunk passed
          +1 javadoc 0m 34s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 0m 43s the patch passed
          +1 compile 1m 33s the patch passed
          +1 javac 1m 33s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 mvnsite 0m 46s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 21s the patch passed
          +1 javadoc 0m 29s the patch passed
          +1 unit 2m 46s hadoop-mapreduce-client-core in the patch passed.
          -1 unit 126m 46s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          154m 46s



          Reason Tests
          Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849359/MAPREDUCE-6829.005.patch
          JIRA Issue MAPREDUCE-6829
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2c68b9684c85 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / abedb8a
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6880/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6880/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6880/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6880/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 12m 37s trunk passed +1 compile 1m 35s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 10s trunk passed +1 javadoc 0m 34s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 43s the patch passed +1 compile 1m 33s the patch passed +1 javac 1m 33s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 46s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 21s the patch passed +1 javadoc 0m 29s the patch passed +1 unit 2m 46s hadoop-mapreduce-client-core in the patch passed. -1 unit 126m 46s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 154m 46s Reason Tests Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849359/MAPREDUCE-6829.005.patch JIRA Issue MAPREDUCE-6829 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2c68b9684c85 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / abedb8a Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6880/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6880/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6880/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6880/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          I think org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl timeout is not related to the change.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - I think org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl timeout is not related to the change.
          Hide
          kasha Karthik Kambatla added a comment -

          +1. Checking this in..

          Show
          kasha Karthik Kambatla added a comment - +1. Checking this in..
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for adding this improvement, Miklos Szegedi. Just committed this to trunk and branch-2.

          Show
          kasha Karthik Kambatla added a comment - Thanks for adding this improvement, Miklos Szegedi . Just committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11179 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11179/)
          MAPREDUCE-6829. Add peak memory usage counter for each task. (Miklos (kasha: rev c65f884fc7e08118524f8c88737119d8196b4c1b)

          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/org/apache/hadoop/mapreduce/TaskCounter.properties
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskCounter.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11179 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11179/ ) MAPREDUCE-6829 . Add peak memory usage counter for each task. (Miklos (kasha: rev c65f884fc7e08118524f8c88737119d8196b4c1b) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/org/apache/hadoop/mapreduce/TaskCounter.properties (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/TaskCounter.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Thank you, Karthik Kambatla!

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you, Karthik Kambatla !
          Hide
          mingma Ming Ma added a comment -

          With YARN-3045, is it still necessary? Container level metrics like this seems to be quite useful for other frameworks other than MR and it is something YARN can provide if it hasn't been done.

          Show
          mingma Ming Ma added a comment - With YARN-3045 , is it still necessary? Container level metrics like this seems to be quite useful for other frameworks other than MR and it is something YARN can provide if it hasn't been done.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Thank you for the comment, Ming Ma. This jira primarily targeted branch-2. I agree that ATS should be the primary source of this information in versions 3 and above. Just as a side note, it was important to separate map and reduce numbers. Is this possible with YARN-3045?

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you for the comment, Ming Ma . This jira primarily targeted branch-2. I agree that ATS should be the primary source of this information in versions 3 and above. Just as a side note, it was important to separate map and reduce numbers. Is this possible with YARN-3045 ?
          Hide
          mingma Ming Ma added a comment -

          Miklos Szegedi if each MR task can somehow record the reference to its container id, then end users can get the data via taskId -> containerId -> containerUsage. Sure such approach is only useful when we expect more container metrics at the yarn layer will be added thus frameworks like MR can get the new metrics automatically.

          Show
          mingma Ming Ma added a comment - Miklos Szegedi if each MR task can somehow record the reference to its container id, then end users can get the data via taskId -> containerId -> containerUsage. Sure such approach is only useful when we expect more container metrics at the yarn layer will be added thus frameworks like MR can get the new metrics automatically.
          Hide
          jlowe Jason Lowe added a comment -

          I reverted this from branch-2 due to the backward incompatibility that can manifest during rolling upgrades while a job is running. See the discussion in MAPREDUCE-6922.

          Show
          jlowe Jason Lowe added a comment - I reverted this from branch-2 due to the backward incompatibility that can manifest during rolling upgrades while a job is running. See the discussion in MAPREDUCE-6922 .

            People

            • Assignee:
              miklos.szegedi@cloudera.com Miklos Szegedi
              Reporter:
              yufeigu Yufei Gu
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development