Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2104

Rumen TraceBuilder Does Not Emit CPU/Memory Usage Details in Traces

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: tools/rumen
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Adds cpu, physical memory, virtual memory and heap usages to TraceBuilder's output.
    • Tags:
      rumen resource-usage

      Description

      Via MAPREDUCE-220, we now have CPU/Memory usage information in MapReduce JobHistory files. However, Rumen's TraceBuilder
      does not emit this information in the JSON traces. Without this information, GridMix3 cannot emulate CPU/Memory usage correctly.

      1. mapreduce-2104-v1.1.patch
        6 kB
        Amar Kamat
      2. mapreduce-2104-v1.7.patch
        16 kB
        Amar Kamat
      3. mapreduce-2104-v1.8.1.patch
        22 kB
        Amar Kamat

        Issue Links

          Activity

          Hide
          Ranjit Mathew added a comment -

          MAPREDUCE-220 provides the basic metrics for CPU/Memory usage of MapReduce Tasks.

          Show
          Ranjit Mathew added a comment - MAPREDUCE-220 provides the basic metrics for CPU/Memory usage of MapReduce Tasks.
          Hide
          Ranjit Mathew added a comment -

          MAPREDUCE-1702 needs this feature in Rumen to be able to emulate CPU/Memory usage.

          Show
          Ranjit Mathew added a comment - MAPREDUCE-1702 needs this feature in Rumen to be able to emulate CPU/Memory usage.
          Hide
          Amar Kamat added a comment -

          Attaching a patch that extracts CPU/Memory usage information from task counters and adds it to the trace file generated by Rumen. test-patch and ant tests (except TestSequenceFileInputFormat) passed on my box.

          Show
          Amar Kamat added a comment - Attaching a patch that extracts CPU/Memory usage information from task counters and adds it to the trace file generated by Rumen. test-patch and ant tests (except TestSequenceFileInputFormat ) passed on my box.
          Hide
          Ravi Gummadi added a comment -

          Some comments on the patch:

          (1) Why are the variable names and method names start with the word "expected" ? Since these are the values that are already seen(and not the values expected), please remove the word "expected" everywhere.

          (2) You missed making changes to HadoopLogAnalyzer.incorporateCounters().

          (3) You missed making changes to LoggedTaskAttempt.deepCompare().

          (4) Empty comments as javadoc are seen for the new methods. Please add some comments there.

          (5) Better to change the new(and may be existing ones also ?) data members of LoggedTaskAttempt private and use setters and getters everywhere because even if I change getExpectedVirtualMemorySize() to return expectedVirtualMemorySizeInBytes-2, testcase is not failing.

          Show
          Ravi Gummadi added a comment - Some comments on the patch: (1) Why are the variable names and method names start with the word "expected" ? Since these are the values that are already seen(and not the values expected), please remove the word "expected" everywhere. (2) You missed making changes to HadoopLogAnalyzer.incorporateCounters(). (3) You missed making changes to LoggedTaskAttempt.deepCompare(). (4) Empty comments as javadoc are seen for the new methods. Please add some comments there. (5) Better to change the new(and may be existing ones also ?) data members of LoggedTaskAttempt private and use setters and getters everywhere because even if I change getExpectedVirtualMemorySize() to return expectedVirtualMemorySizeInBytes-2, testcase is not failing.
          Hide
          Amar Kamat added a comment -

          Attaching a new patch. test-patch and ant-tests passed on my box. This patch extracts resource usage metric values (like cpu-usage, physical memory, virtual memory and heap usage) into a separate class and provides setters/getters to access it.

          Show
          Amar Kamat added a comment - Attaching a new patch. test-patch and ant-tests passed on my box. This patch extracts resource usage metric values (like cpu-usage, physical memory, virtual memory and heap usage) into a separate class and provides setters/getters to access it.
          Hide
          Amar Kamat added a comment -

          Attaching a modified patch adding this feature to HadoopLogsAnalyzer. Modified the testcase too. test-patch and unit tests passed.

          Show
          Amar Kamat added a comment - Attaching a modified patch adding this feature to HadoopLogsAnalyzer. Modified the testcase too. test-patch and unit tests passed.
          Hide
          Amar Kamat added a comment -

          Running through Hudson.

          Show
          Amar Kamat added a comment - Running through Hudson.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481438/mapreduce-2104-v1.8.1.patch
          against trunk revision 1131265.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/347//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/347//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/347//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481438/mapreduce-2104-v1.8.1.patch against trunk revision 1131265. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/347//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/347//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/347//console This message is automatically generated.
          Hide
          Amar Kamat added a comment -

          The findbugs issue and the unit test failure are not related to this patch.

          Show
          Amar Kamat added a comment - The findbugs issue and the unit test failure are not related to this patch.
          Hide
          Ravi Gummadi added a comment -

          Patch looks fine to me.
          +1

          Show
          Ravi Gummadi added a comment - Patch looks fine to me. +1
          Hide
          Amar Kamat added a comment -

          I just committed the latest patch to trunk. Thanks Ravi for the review!

          Show
          Amar Kamat added a comment - I just committed the latest patch to trunk. Thanks Ravi for the review!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #715 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/715/)
          MAPREDUCE-2104. [Rumen] Add Cpu, Memory and Heap usages to TraceBuilder's output. (amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132529
          Files :

          • /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java
          • /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/HadoopLogsAnalyzer.java
          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
          • /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/TaskInfo.java
          • /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/ResourceUsageMetrics.java
          • /hadoop/mapreduce/trunk/src/test/tools/data/rumen/small-trace-test/v20-resource-usage-log.gz
          • /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/ZombieJob.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #715 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/715/ ) MAPREDUCE-2104 . [Rumen] Add Cpu, Memory and Heap usages to TraceBuilder's output. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132529 Files : /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/HadoopLogsAnalyzer.java /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/TaskInfo.java /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/ResourceUsageMetrics.java /hadoop/mapreduce/trunk/src/test/tools/data/rumen/small-trace-test/v20-resource-usage-log.gz /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/rumen/ZombieJob.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Ranjit Mathew
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development