Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3118

Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Backports latest features from trunk to 0.20.206 branch.
    • Tags:
      gridmix rumen backport

      Description

      Backporting all the features and bugfixes that went into gridmix and rumen of trunk to hadoop 0.20 security branch. This will enable using all these gridmix features and run gridmix/rumen on the history logs of 0.20 security branch.

      1. gridmix_rumen_backports.v2.4.patch
        611 kB
        Ravi Gummadi
      2. gridmix_rumen_backports.v2.5.patch
        614 kB
        Ravi Gummadi
      3. gridmix_rumen_backports.v2.6.patch
        614 kB
        Ravi Gummadi

        Activity

        Hide
        Ravi Gummadi added a comment -

        System tests patches are not part of this for now. They will be added by Vinay (may be later).

        Backporting patches (features and bugfixes) are from the following JIRAs:

        MAPREDUCE-2596: [Gridmix] Summarize Gridmix runs.
        MAPREDUCE-2107: [Gridmix] Total heap usage emulation in Gridmix.
        MAPREDUCE-2106: [Gridmix] Cumulative CPU usage emulation in Gridmix.
        MAPREDUCE-2543: [Gridmix] High-Ram feature emulation testcase.
        MAPREDUCE-2408: [Gridmix] Compression emulation in Gridmix.
        MAPREDUCE-2137: Provide mapping between jobs of trace file and the corresponding simulated cluster's jobs in Gridmix.
        MAPREDUCE-2407: Make GridMix emulate usage of distributed cache files in simulated jobs.
        MAPREDUCE-2416: Remove the restriction of specifying group names in users-list file for Gridmix in RoundRobinUserResolver mode.
        MAPREDUCE-1931: Gridmix documentation.
        MAPREDUCE-2417: Fix Gridmix in RoundRobinUserResolver mode to map testing/proxy users to unique users in a trace.
        MAPREDUCE-2095: Fix Gridmix to run from compressed traces.
        MAPREDUCE-1989: Fix error message in gridmix when user resolver is set and no user list is given.
        MAPREDUCE-1979: Fix 'Output directory already exists' error in gridmix when gridmix.output.directory is not defined.
        MAPREDUCE-1975: Fix unnecessary InterruptedException log in gridmix.


        MAPREDUCE-2078: Fix TraceBuilder to generate traces when a globbed job history path is given.
        MAPREDUCE-2153: Rumen should bring in job configuration properties into the trace file.
        MAPREDUCE-2104: [Rumen] Add Cpu, Memory and Heap usages to TraceBuilder's output.
        MAPREDUCE-1978: Rumen TraceBuilder should provide recursive input folder scanning.
        MAPREDUCE-1918: Add documentation to Rumen.
        MAPREDUCE-2000: Rumen is not able to extract counters for Job history logs from Hadoop 0.20.
        MAPREDUCE-1982: Fix Rumen's TraceBuilder to extract job name from either of configuration properties mapreduce.job.name and mapred.job.name.
        MAPREDUCE-1925: Fix failing TestRumenJobTraces.

        This backporting also needs minor changes to mapreduce task progress reporting because some of these gridmix features rely on the task progress values.

        Show
        Ravi Gummadi added a comment - System tests patches are not part of this for now. They will be added by Vinay (may be later). Backporting patches (features and bugfixes) are from the following JIRAs: MAPREDUCE-2596 : [Gridmix] Summarize Gridmix runs. MAPREDUCE-2107 : [Gridmix] Total heap usage emulation in Gridmix. MAPREDUCE-2106 : [Gridmix] Cumulative CPU usage emulation in Gridmix. MAPREDUCE-2543 : [Gridmix] High-Ram feature emulation testcase. MAPREDUCE-2408 : [Gridmix] Compression emulation in Gridmix. MAPREDUCE-2137 : Provide mapping between jobs of trace file and the corresponding simulated cluster's jobs in Gridmix. MAPREDUCE-2407 : Make GridMix emulate usage of distributed cache files in simulated jobs. MAPREDUCE-2416 : Remove the restriction of specifying group names in users-list file for Gridmix in RoundRobinUserResolver mode. MAPREDUCE-1931 : Gridmix documentation. MAPREDUCE-2417 : Fix Gridmix in RoundRobinUserResolver mode to map testing/proxy users to unique users in a trace. MAPREDUCE-2095 : Fix Gridmix to run from compressed traces. MAPREDUCE-1989 : Fix error message in gridmix when user resolver is set and no user list is given. MAPREDUCE-1979 : Fix 'Output directory already exists' error in gridmix when gridmix.output.directory is not defined. MAPREDUCE-1975 : Fix unnecessary InterruptedException log in gridmix. MAPREDUCE-2078 : Fix TraceBuilder to generate traces when a globbed job history path is given. MAPREDUCE-2153 : Rumen should bring in job configuration properties into the trace file. MAPREDUCE-2104 : [Rumen] Add Cpu, Memory and Heap usages to TraceBuilder's output. MAPREDUCE-1978 : Rumen TraceBuilder should provide recursive input folder scanning. MAPREDUCE-1918 : Add documentation to Rumen. MAPREDUCE-2000 : Rumen is not able to extract counters for Job history logs from Hadoop 0.20. MAPREDUCE-1982 : Fix Rumen's TraceBuilder to extract job name from either of configuration properties mapreduce.job.name and mapred.job.name. MAPREDUCE-1925 : Fix failing TestRumenJobTraces. This backporting also needs minor changes to mapreduce task progress reporting because some of these gridmix features rely on the task progress values.
        Hide
        Ravi Gummadi added a comment -

        Attaching patch that incorporates all the features and bugfixes mentioned in previous comment.

        Show
        Ravi Gummadi added a comment - Attaching patch that incorporates all the features and bugfixes mentioned in previous comment.
        Hide
        Ravi Gummadi added a comment -

        Attaching new patch adding src.test.data to build-contrib.xml (which is missing in 20-security branch) to make TestGridmixSubmission to run properly.

        Show
        Ravi Gummadi added a comment - Attaching new patch adding src.test.data to build-contrib.xml (which is missing in 20-security branch) to make TestGridmixSubmission to run properly.
        Hide
        Matt Foley added a comment -

        Hi Ravi, it will be great to see these improvements in 0.20.206. Thanks.

        Show
        Matt Foley added a comment - Hi Ravi, it will be great to see these improvements in 0.20.206. Thanks.
        Hide
        Amar Kamat added a comment -

        @Matt I have taken up the review of the latest patch.

        @All Vinay has tested this patch on a test cluster and seems like all the latest Gridmix features work fine. Vinay also backported the system tests he wrote for trunk to make sure that all the features are tested thoroughly.

        Show
        Amar Kamat added a comment - @Matt I have taken up the review of the latest patch. @All Vinay has tested this patch on a test cluster and seems like all the latest Gridmix features work fine. Vinay also backported the system tests he wrote for trunk to make sure that all the features are tested thoroughly.
        Hide
        Amar Kamat added a comment -

        I see some MRJobConfig fields in DistributedCacheEmulator's javadoc. Can this be changed to their new locations maybe in JobContext or DistributedCache?
        Rest of the patch looks good to me.

        @Ravi: Can you upload test-patch and ant-test status along with the final patch?

        Show
        Amar Kamat added a comment - I see some MRJobConfig fields in DistributedCacheEmulator's javadoc. Can this be changed to their new locations maybe in JobContext or DistributedCache? Rest of the patch looks good to me. @Ravi: Can you upload test-patch and ant-test status along with the final patch?
        Hide
        Ravi Gummadi added a comment -

        Incorporated Amar's review comments.

        Unit tests and test-patch passed on my local machine.

        Show
        Ravi Gummadi added a comment - Incorporated Amar's review comments. Unit tests and test-patch passed on my local machine.
        Hide
        Amar Kamat added a comment -

        Folks,
        This patch adds a getProgress() API to public classes and interfaces like:
        1. Progress.java
        2. TaskInputOutputContext.java
        3. Reporter.java

        This API provides the task's current progress and is immensely useful of Gridmix. We tend to believe that this might get classified as a backward incompatible change. We want to be sure about the incompatibility side of the story and make a call accordingly.

        Kindly let us know your thoughts/comments regarding the same.

        Show
        Amar Kamat added a comment - Folks, This patch adds a getProgress() API to public classes and interfaces like: 1. Progress.java 2. TaskInputOutputContext.java 3. Reporter.java This API provides the task's current progress and is immensely useful of Gridmix. We tend to believe that this might get classified as a backward incompatible change. We want to be sure about the incompatibility side of the story and make a call accordingly. Kindly let us know your thoughts/comments regarding the same.
        Hide
        Ravi Gummadi added a comment -

        Please come back if there are any issues. We are planning to commit this soon.

        Show
        Ravi Gummadi added a comment - Please come back if there are any issues. We are planning to commit this soon.
        Hide
        Amar Kamat added a comment -

        Folks,
        We ran the wordcount and Gridmix3 from Hadoop 0.20.205 on a Hadoop cluster running a MR-3118 patched version of 0.20.205. All the jobs ran fine. We didn't see any compatibility issues. I will commit this patch.

        Show
        Amar Kamat added a comment - Folks, We ran the wordcount and Gridmix3 from Hadoop 0.20.205 on a Hadoop cluster running a MR-3118 patched version of 0.20.205. All the jobs ran fine. We didn't see any compatibility issues. I will commit this patch.
        Hide
        Amar Kamat added a comment -

        I just committed this to the 0.20.206 branch. Great job Ravi! Thanks Vinay for helping with the testing.

        Show
        Amar Kamat added a comment - I just committed this to the 0.20.206 branch. Great job Ravi! Thanks Vinay for helping with the testing.
        Hide
        Matt Foley added a comment -

        Closed upon release of Hadoop-1.1.0.

        Show
        Matt Foley added a comment - Closed upon release of Hadoop-1.1.0.

          People

          • Assignee:
            Ravi Gummadi
            Reporter:
            Ravi Gummadi
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development