Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/gridmix
    • Labels:
      None
    • Release Note:
      Adds cumulative cpu usage emulation to Gridmix

      Description

      MAPREDUCE-220 makes CPU/Memory usage of Tasks available in JobHistory files. Use this
      to emulate the CPU usage of Tasks (of course, once MAPREDUCE-2104 is done).

        Issue Links

          Activity

          Hide
          Amar Kamat added a comment -

          Following are the bugs that were discovered while working in this issue. Will incorporate them in this issue itself.

          Show
          Amar Kamat added a comment - Following are the bugs that were discovered while working in this issue. Will incorporate them in this issue itself.
          Hide
          Amar Kamat added a comment -

          Attaching a patch to emulate the CPU usage in GridMix. This patch also adds the new resource emulation framework to GridMix. Updated the GridMix documentation/manual regarding the same. test-patch passed. GridMix tests passed.

          Show
          Amar Kamat added a comment - Attaching a patch to emulate the CPU usage in GridMix. This patch also adds the new resource emulation framework to GridMix. Updated the GridMix documentation/manual regarding the same. test-patch passed. GridMix tests passed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481692/gridmix-cpu-emulation-v1.16-1.patch
          against trunk revision 1132807.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/356//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/356//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/356//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481692/gridmix-cpu-emulation-v1.16-1.patch against trunk revision 1132807. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/356//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/356//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/356//console This message is automatically generated.
          Hide
          Amar Kamat added a comment -

          Attaching a new patch. Changes are as follows
          1. Cpu usage emulation is weighed
          2. Added inline match calls
          3. Added status reporting to avoid task kills due to lack of status updates
          4. Modified the test case to cover more testcase scenarios.

          test-patch and gridmix tests passed on my box.

          Note: This patch is generated for http://svn.apache.org/repos/asf/hadoop/common/trunk/mapreduce.

          Show
          Amar Kamat added a comment - Attaching a new patch. Changes are as follows 1. Cpu usage emulation is weighed 2. Added inline match calls 3. Added status reporting to avoid task kills due to lack of status updates 4. Modified the test case to cover more testcase scenarios. test-patch and gridmix tests passed on my box. Note: This patch is generated for http://svn.apache.org/repos/asf/hadoop/common/trunk/mapreduce .
          Hide
          Vinay Kumar Thota added a comment -

          Patch reviewed and tested. It looks good to me.

          +1.

          Show
          Vinay Kumar Thota added a comment - Patch reviewed and tested. It looks good to me. +1.
          Hide
          Amar Kamat added a comment -

          I just committed the latest patch to trunk. test-patch and gridmix tests passed on my box. Thanks Ravi for testing this.

          Show
          Amar Kamat added a comment - I just committed the latest patch to trunk. test-patch and gridmix tests passed on my box. Thanks Ravi for testing this.
          Hide
          Amar Kamat added a comment -

          Thanks Vinay for testing this!

          Show
          Amar Kamat added a comment - Thanks Vinay for testing this!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #726 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/726/)
          MAPREDUCE-2106. [Gridmix] Cumulative CPU usage emulation in Gridmix. (amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135396
          Files :

          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/LoadJob.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixRecord.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/LoadSplit.java
          • /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/gridmix.xml
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/CumulativeCpuUsageEmulatorPlugin.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageEmulatorPlugin.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageMatcher.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Progressive.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GridmixKey.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #726 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/726/ ) MAPREDUCE-2106 . [Gridmix] Cumulative CPU usage emulation in Gridmix. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135396 Files : /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/LoadJob.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixRecord.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/LoadSplit.java /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/gridmix.xml /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/CumulativeCpuUsageEmulatorPlugin.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageEmulatorPlugin.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageMatcher.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Progressive.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GridmixKey.java

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Ranjit Mathew
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development