Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/gridmix
    • Labels:
      None
    • Release Note:
      Hide
      Adds total heap usage emulation to Gridmix. Also, Gridmix can configure the simulated task's JVM heap options with max heap options obtained from the original task (via Rumen). Use 'gridmix.task.jvm-options.enable' to disable the task max heap options configuration.
      Show
      Adds total heap usage emulation to Gridmix. Also, Gridmix can configure the simulated task's JVM heap options with max heap options obtained from the original task (via Rumen). Use 'gridmix.task.jvm-options.enable' to disable the task max heap options configuration.
    • Tags:
      gridmix heap-usage

      Description

      MAPREDUCE-220 makes CPU/Memory usage of Tasks available in JobHistory files. Use this
      to emulate the memory usage of Tasks (of course, once MAPREDUCE-2104 is done).

        Issue Links

          Activity

          Hide
          Ravi Gummadi added a comment -

          Am planning to emulate committed heap usage of the task JVM as a first step twards emulating memory usage. This committed heap usage comes from MAPREDUCE-2469 into job history file and then MAPREDUCE-2104 will bring that info into the trace file.

          Show
          Ravi Gummadi added a comment - Am planning to emulate committed heap usage of the task JVM as a first step twards emulating memory usage. This committed heap usage comes from MAPREDUCE-2469 into job history file and then MAPREDUCE-2104 will bring that info into the trace file.
          Hide
          Amar Kamat added a comment -

          Attaching a patch the adds total-heap-usage emulation to Gridmix. The goal is to do a very basic heap usage emulation and using this tool to exert some sort of memory load on the simulated task's JVM. This patch also sets the simulated task's heap options extracted from the original job obtained via Rumen.

          test-patch and gridmix junit tests passed.

          Show
          Amar Kamat added a comment - Attaching a patch the adds total-heap-usage emulation to Gridmix. The goal is to do a very basic heap usage emulation and using this tool to exert some sort of memory load on the simulated task's JVM. This patch also sets the simulated task's heap options extracted from the original job obtained via Rumen. test-patch and gridmix junit tests passed.
          Hide
          Vinay Kumar Thota added a comment -

          Patch reviewed and tested. It looks good to me.
          +1.

          Show
          Vinay Kumar Thota added a comment - Patch reviewed and tested. It looks good to me. +1.
          Hide
          Amar Kamat added a comment -

          I just committed the latest patch to trunk. Thanks Ravi and Vinay!

          Show
          Amar Kamat added a comment - I just committed the latest patch to trunk. Thanks Ravi and Vinay!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #728 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/728/)
          MAPREDUCE-2107. [Gridmix] Total heap usage emulation in Gridmix. (Amar Kamat and Ravi Gummadi via amarrk)

          amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1136000
          Files :

          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GridmixJob.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestHighRamJob.java
          • /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/gridmix.xml
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixMemoryEmulation.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/CumulativeCpuUsageEmulatorPlugin.java
          • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #728 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/728/ ) MAPREDUCE-2107 . [Gridmix] Total heap usage emulation in Gridmix. (Amar Kamat and Ravi Gummadi via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1136000 Files : /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GridmixJob.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestHighRamJob.java /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/gridmix.xml /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixMemoryEmulation.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/CumulativeCpuUsageEmulatorPlugin.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Ranjit Mathew
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development