Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2469

Task counters should also report the total heap usage of the task

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: task
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Task attempt's total heap usage gets recorded and published via counters as COMMITTED_HEAP_BYTES.
    • Tags:
      total-heap, task-counters

      Description

      Currently, the task counters report VSS and RSS usage of the task. The task counter should also report the total heap usage of the task also. The task might be configured with a max heap size of M but the task's total heap usage might only be H, where H < M. In such a case, knowing only M doesn't provide a complete picture of the task's memory usage.

      1. mapreduce-2469-v1.4.patch
        13 kB
        Amar Kamat
      2. mapreduce-2469-v1.3.patch
        13 kB
        Amar Kamat
      3. mapreduce-2469-v1.2.patch
        13 kB
        Amar Kamat
      4. mapreduce-2469-v1.1.patch
        12 kB
        Amar Kamat
      5. 2469.v0.1.patch
        2 kB
        Ravi Gummadi

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          2d 21h 26m 3 Amar Kamat 01/Jun/11 15:09
          Open Open Patch Available Patch Available
          26d 12h 19m 4 Amar Kamat 01/Jun/11 15:21
          Patch Available Patch Available Resolved Resolved
          12h 54m 1 Amar Kamat 02/Jun/11 04:16
          Resolved Resolved Closed Closed
          165d 21h 32m 1 Arun C Murthy 15/Nov/11 00:49
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Amar Kamat made changes -
          Link This issue relates to MAPREDUCE-2777 [ MAPREDUCE-2777 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #708 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/708/)
          MAPREDUCE-2469. Task counters should also report the total heap usage of the task. (Ravi Gummadi and Amar Ramesh Kamat via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130393
          Files :

          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/TaskCounter.java
          • /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/TestJobCounters.java
          • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #708 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/708/ ) MAPREDUCE-2469 . Task counters should also report the total heap usage of the task. (Ravi Gummadi and Amar Ramesh Kamat via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130393 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/Task.java /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/TaskCounter.java /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/TestJobCounters.java /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties
          Amar Kamat made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Release Note Task attempt's total heap usage gets recorded and published via counters as COMMITTED_HEAP_BYTES.
          Fix Version/s 0.23.0 [ 12315570 ]
          Resolution Fixed [ 1 ]
          Hide
          Amar Kamat added a comment -

          I just committed the latest patch to trunk. Thanks Ravi, Santosh and Owen!

          Show
          Amar Kamat added a comment - I just committed the latest patch to trunk. Thanks Ravi, Santosh and Owen!
          Hide
          Owen O'Malley added a comment -

          +1

          Show
          Owen O'Malley added a comment - +1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481091/mapreduce-2469-v1.4.patch
          against trunk revision 1130147.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481091/mapreduce-2469-v1.4.patch against trunk revision 1130147. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/329//console This message is automatically generated.
          Amar Kamat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amar Kamat made changes -
          Attachment mapreduce-2469-v1.4.patch [ 12481091 ]
          Hide
          Amar Kamat added a comment -

          Attaching a new patch incorporating Ravi's offline comments. test-patch passed. The only change is in TestJobCounters and that passed on my box.

          Show
          Amar Kamat added a comment - Attaching a new patch incorporating Ravi's offline comments. test-patch passed. The only change is in TestJobCounters and that passed on my box.
          Amar Kamat made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Amar Kamat added a comment -

          The testcase failure (Test#7, Archive: Deleting a file in archive) and the findbugs failures are not related to this patch.

          Show
          Amar Kamat added a comment - The testcase failure (Test#7, Archive: Deleting a file in archive) and the findbugs failures are not related to this patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481044/mapreduce-2469-v1.3.patch
          against trunk revision 1129771.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481044/mapreduce-2469-v1.3.patch against trunk revision 1129771. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/326//console This message is automatically generated.
          Amar Kamat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amar Kamat made changes -
          Attachment mapreduce-2469-v1.3.patch [ 12481044 ]
          Hide
          Amar Kamat added a comment -

          Attaching a new patch that makes the testcase's memory loader static so that once the map class gets loaded, the loader stays in the jvm's heap without getting GCed.

          Show
          Amar Kamat added a comment - Attaching a new patch that makes the testcase's memory loader static so that once the map class gets loaded, the loader stays in the jvm's heap without getting GCed.
          Amar Kamat made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Amar Kamat added a comment -

          TestJobCounters passed on my box. My guess is that the JVM is releasing the memory to the OS after the GC, causing Runtime.getRuntime().totalMemory() to decrease before it gets recorded in the task's counter. Its very much dependent on the kernel. Some linux kernels dont support this feature (including the one which I am using). I will debug this further.

          Show
          Amar Kamat added a comment - TestJobCounters passed on my box. My guess is that the JVM is releasing the memory to the OS after the GC, causing Runtime.getRuntime().totalMemory() to decrease before it gets recorded in the task's counter. Its very much dependent on the kernel. Some linux kernels dont support this feature (including the one which I am using). I will debug this further.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480900/mapreduce-2469-v1.2.patch
          against trunk revision 1128394.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.mapred.TestJobCounters

          -1 contrib tests. The patch failed contrib unit tests.

          -1 system test framework. The patch failed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480900/mapreduce-2469-v1.2.patch against trunk revision 1128394. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestJobCounters -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/321//console This message is automatically generated.
          Amar Kamat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Amar Kamat added a comment -

          Running through Hudson. The only change is in TaskCounter.properties.

          Show
          Amar Kamat added a comment - Running through Hudson. The only change is in TaskCounter.properties.
          Amar Kamat made changes -
          Attachment mapreduce-2469-v1.2.patch [ 12480900 ]
          Hide
          Amar Kamat added a comment -

          Attaching a new patch. Changed TaskCounter.properties to provide textual description to the newly added counter. Thanks Owen for pointing it out.

          test-patch passed on my box. Changes to TaskCounter.properties should not affect the test cases.

          Show
          Amar Kamat added a comment - Attaching a new patch. Changed TaskCounter.properties to provide textual description to the newly added counter. Thanks Owen for pointing it out. test-patch passed on my box. Changes to TaskCounter.properties should not affect the test cases.
          Amar Kamat made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Owen O'Malley added a comment -

          The patch looks good. I don't see how the patch could be causing the testcase failures or the findbugs warnings.

          Show
          Owen O'Malley added a comment - The patch looks good. I don't see how the patch could be causing the testcase failures or the findbugs warnings.
          Amar Kamat made changes -
          Link This issue supercedes MAPREDUCE-2104 [ MAPREDUCE-2104 ]
          Amar Kamat made changes -
          Link This issue is blocked by MAPREDUCE-2104 [ MAPREDUCE-2104 ]
          Amar Kamat made changes -
          Link This issue is blocked by MAPREDUCE-2104 [ MAPREDUCE-2104 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480766/mapreduce-2469-v1.1.patch
          against trunk revision 1128394.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.tools.TestHadoopArchives
          org.apache.hadoop.tools.TestHarFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480766/mapreduce-2469-v1.1.patch against trunk revision 1128394. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.tools.TestHadoopArchives org.apache.hadoop.tools.TestHarFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//console This message is automatically generated.
          Amar Kamat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Amar Kamat added a comment -

          Running through Hudson. The testcase sets the Xms and Xmx value of the map/reduce tasks and then loads the task's memory to the configured target value.

          Show
          Amar Kamat added a comment - Running through Hudson. The testcase sets the Xms and Xmx value of the map/reduce tasks and then loads the task's memory to the configured target value.
          Amar Kamat made changes -
          Attachment mapreduce-2469-v1.1.patch [ 12480766 ]
          Hide
          Amar Kamat added a comment -

          Attaching a patch that adds a new counter, COMMITTED_HEAP_BYTES, to task counters. This counter records the total committed heap usage on the task. Added a testcase to test the same. test-patch and ant-tests passed on my box.

          Show
          Amar Kamat added a comment - Attaching a patch that adds a new counter, COMMITTED_HEAP_BYTES, to task counters. This counter records the total committed heap usage on the task. Added a testcase to test the same. test-patch and ant-tests passed on my box.
          Amar Kamat made changes -
          Assignee Ravi Gummadi [ ravidotg ] Amar Kamat [ amar_kamat ]
          Hide
          Santosh Kumar added a comment -

          I will be taking up this Jira.

          Show
          Santosh Kumar added a comment - I will be taking up this Jira.
          Hide
          Amar Kamat added a comment -

          The patch looks fine to me. For the test case, can we do the following
          1. Start a Mini-MR cluster.
          2. Submit a no-op job (say job1) with 1 map and 1 reduce
          3. Submit a word count like job (say job2) with 1 map and 1 reduce.
          4. Check if job1's heap usage (reported via its counters) is less than that of job2.

          For (3), any job that can add some extra load in the task's heap should suffice. Also both the jobs should have 'mapred.child.java.opts' having -Xms set to a very low value.

          Thoughts?

          Show
          Amar Kamat added a comment - The patch looks fine to me. For the test case, can we do the following 1. Start a Mini-MR cluster. 2. Submit a no-op job (say job1) with 1 map and 1 reduce 3. Submit a word count like job (say job2) with 1 map and 1 reduce. 4. Check if job1's heap usage (reported via its counters) is less than that of job2. For (3), any job that can add some extra load in the task's heap should suffice. Also both the jobs should have 'mapred.child.java.opts' having -Xms set to a very low value. Thoughts?
          Ravi Gummadi made changes -
          Attachment 2469.v0.1.patch [ 12479437 ]
          Hide
          Ravi Gummadi added a comment -

          Attaching first cut patch. This doesn't have unit test. Working on adding unit test.

          Show
          Ravi Gummadi added a comment - Attaching first cut patch. This doesn't have unit test. Working on adding unit test.
          Hide
          Ravi Gummadi added a comment -

          I am planning to add a new TaskCounter COMMITTED_HEAP_BYTES that represents the total committed heap size of the task at any given time. This is also updated everytime other resource counters are updated.

          Show
          Ravi Gummadi added a comment - I am planning to add a new TaskCounter COMMITTED_HEAP_BYTES that represents the total committed heap size of the task at any given time. This is also updated everytime other resource counters are updated.
          Ravi Gummadi made changes -
          Assignee Ravi Gummadi [ ravidotg ]
          Amar Kamat made changes -
          Field Original Value New Value
          Link This issue blocks MAPREDUCE-2107 [ MAPREDUCE-2107 ]
          Amar Kamat created issue -

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Amar Kamat
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development