Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2469

Task counters should also report the total heap usage of the task

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: task
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Task attempt's total heap usage gets recorded and published via counters as COMMITTED_HEAP_BYTES.
    • Tags:
      total-heap, task-counters

      Description

      Currently, the task counters report VSS and RSS usage of the task. The task counter should also report the total heap usage of the task also. The task might be configured with a max heap size of M but the task's total heap usage might only be H, where H < M. In such a case, knowing only M doesn't provide a complete picture of the task's memory usage.

      1. mapreduce-2469-v1.4.patch
        13 kB
        Amar Kamat
      2. mapreduce-2469-v1.3.patch
        13 kB
        Amar Kamat
      3. mapreduce-2469-v1.2.patch
        13 kB
        Amar Kamat
      4. mapreduce-2469-v1.1.patch
        12 kB
        Amar Kamat
      5. 2469.v0.1.patch
        2 kB
        Ravi Gummadi

        Issue Links

          Activity

          Hide
          Ravi Gummadi added a comment -

          I am planning to add a new TaskCounter COMMITTED_HEAP_BYTES that represents the total committed heap size of the task at any given time. This is also updated everytime other resource counters are updated.

          Show
          Ravi Gummadi added a comment - I am planning to add a new TaskCounter COMMITTED_HEAP_BYTES that represents the total committed heap size of the task at any given time. This is also updated everytime other resource counters are updated.
          Hide
          Ravi Gummadi added a comment -

          Attaching first cut patch. This doesn't have unit test. Working on adding unit test.

          Show
          Ravi Gummadi added a comment - Attaching first cut patch. This doesn't have unit test. Working on adding unit test.
          Hide
          Amar Kamat added a comment -

          The patch looks fine to me. For the test case, can we do the following
          1. Start a Mini-MR cluster.
          2. Submit a no-op job (say job1) with 1 map and 1 reduce
          3. Submit a word count like job (say job2) with 1 map and 1 reduce.
          4. Check if job1's heap usage (reported via its counters) is less than that of job2.

          For (3), any job that can add some extra load in the task's heap should suffice. Also both the jobs should have 'mapred.child.java.opts' having -Xms set to a very low value.

          Thoughts?

          Show
          Amar Kamat added a comment - The patch looks fine to me. For the test case, can we do the following 1. Start a Mini-MR cluster. 2. Submit a no-op job (say job1) with 1 map and 1 reduce 3. Submit a word count like job (say job2) with 1 map and 1 reduce. 4. Check if job1's heap usage (reported via its counters) is less than that of job2. For (3), any job that can add some extra load in the task's heap should suffice. Also both the jobs should have 'mapred.child.java.opts' having -Xms set to a very low value. Thoughts?
          Hide
          Santosh Kumar added a comment -

          I will be taking up this Jira.

          Show
          Santosh Kumar added a comment - I will be taking up this Jira.
          Hide
          Amar Kamat added a comment -

          Attaching a patch that adds a new counter, COMMITTED_HEAP_BYTES, to task counters. This counter records the total committed heap usage on the task. Added a testcase to test the same. test-patch and ant-tests passed on my box.

          Show
          Amar Kamat added a comment - Attaching a patch that adds a new counter, COMMITTED_HEAP_BYTES, to task counters. This counter records the total committed heap usage on the task. Added a testcase to test the same. test-patch and ant-tests passed on my box.
          Hide
          Amar Kamat added a comment -

          Running through Hudson. The testcase sets the Xms and Xmx value of the map/reduce tasks and then loads the task's memory to the configured target value.

          Show
          Amar Kamat added a comment - Running through Hudson. The testcase sets the Xms and Xmx value of the map/reduce tasks and then loads the task's memory to the configured target value.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480766/mapreduce-2469-v1.1.patch
          against trunk revision 1128394.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.tools.TestHadoopArchives
          org.apache.hadoop.tools.TestHarFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480766/mapreduce-2469-v1.1.patch against trunk revision 1128394. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.tools.TestHadoopArchives org.apache.hadoop.tools.TestHarFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          The patch looks good. I don't see how the patch could be causing the testcase failures or the findbugs warnings.

          Show
          Owen O'Malley added a comment - The patch looks good. I don't see how the patch could be causing the testcase failures or the findbugs warnings.
          Hide
          Amar Kamat added a comment -

          Attaching a new patch. Changed TaskCounter.properties to provide textual description to the newly added counter. Thanks Owen for pointing it out.

          test-patch passed on my box. Changes to TaskCounter.properties should not affect the test cases.

          Show
          Amar Kamat added a comment - Attaching a new patch. Changed TaskCounter.properties to provide textual description to the newly added counter. Thanks Owen for pointing it out. test-patch passed on my box. Changes to TaskCounter.properties should not affect the test cases.
          Hide
          Amar Kamat added a comment -

          Running through Hudson. The only change is in TaskCounter.properties.

          Show
          Amar Kamat added a comment - Running through Hudson. The only change is in TaskCounter.properties.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480900/mapreduce-2469-v1.2.patch
          against trunk revision 1128394.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.mapred.TestJobCounters

          -1 contrib tests. The patch failed contrib unit tests.

          -1 system test framework. The patch failed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480900/mapreduce-2469-v1.2.patch against trunk revision 1128394. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestJobCounters -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//console This message is automatically generated.
          Hide
          Amar Kamat added a comment -

          TestJobCounters passed on my box. My guess is that the JVM is releasing the memory to the OS after the GC, causing Runtime.getRuntime().totalMemory() to decrease before it gets recorded in the task's counter. Its very much dependent on the kernel. Some linux kernels dont support this feature (including the one which I am using). I will debug this further.

          Show
          Amar Kamat added a comment - TestJobCounters passed on my box. My guess is that the JVM is releasing the memory to the OS after the GC, causing Runtime.getRuntime().totalMemory() to decrease before it gets recorded in the task's counter. Its very much dependent on the kernel. Some linux kernels dont support this feature (including the one which I am using). I will debug this further.
          Hide
          Amar Kamat added a comment -

          Attaching a new patch that makes the testcase's memory loader static so that once the map class gets loaded, the loader stays in the jvm's heap without getting GCed.

          Show
          Amar Kamat added a comment - Attaching a new patch that makes the testcase's memory loader static so that once the map class gets loaded, the loader stays in the jvm's heap without getting GCed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481044/mapreduce-2469-v1.3.patch
          against trunk revision 1129771.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481044/mapreduce-2469-v1.3.patch against trunk revision 1129771. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//console This message is automatically generated.
          Hide
          Amar Kamat added a comment -

          The testcase failure (Test#7, Archive: Deleting a file in archive) and the findbugs failures are not related to this patch.

          Show
          Amar Kamat added a comment - The testcase failure (Test#7, Archive: Deleting a file in archive) and the findbugs failures are not related to this patch.
          Hide
          Amar Kamat added a comment -

          Attaching a new patch incorporating Ravi's offline comments. test-patch passed. The only change is in TestJobCounters and that passed on my box.

          Show
          Amar Kamat added a comment - Attaching a new patch incorporating Ravi's offline comments. test-patch passed. The only change is in TestJobCounters and that passed on my box.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481091/mapreduce-2469-v1.4.patch
          against trunk revision 1130147.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481091/mapreduce-2469-v1.4.patch against trunk revision 1130147. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          +1

          Show
          Owen O'Malley added a comment - +1
          Hide
          Amar Kamat added a comment -

          I just committed the latest patch to trunk. Thanks Ravi, Santosh and Owen!

          Show
          Amar Kamat added a comment - I just committed the latest patch to trunk. Thanks Ravi, Santosh and Owen!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #708 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/708/)
          MAPREDUCE-2469. Task counters should also report the total heap usage of the task. (Ravi Gummadi and Amar Ramesh Kamat via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130393
          Files :

          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/TaskCounter.java
          • /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/TestJobCounters.java
          • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #708 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/708/ ) MAPREDUCE-2469 . Task counters should also report the total heap usage of the task. (Ravi Gummadi and Amar Ramesh Kamat via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130393 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/Task.java /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/TaskCounter.java /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/TestJobCounters.java /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Amar Kamat
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development