Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: contrib/gridmix
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:

      • add (random) location information for each sleep map task.
      • make the parameters used in stress submission load throttling configurable.
      1. mr-1936-yhadoop-20.1xx.patch
        36 kB
        Hong Tang
      2. mr-1936-20100715.patch
        36 kB
        Hong Tang
      3. mr-1936-20100720.patch
        39 kB
        Hong Tang
      4. mr-1936-delta-20.1xx.patch
        8 kB
        Hong Tang

        Activity

        Hide
        Hong Tang added a comment -

        Patch for yhadoop-20.1xx branch. Not to be committed.

        Show
        Hong Tang added a comment - Patch for yhadoop-20.1xx branch. Not to be committed.
        Hide
        Hong Tang added a comment -

        The patch also incorporates the changes in MAPREDUCE-1833.

        Show
        Hong Tang added a comment - The patch also incorporates the changes in MAPREDUCE-1833 .
        Hide
        Hong Tang added a comment -

        test-patch passes on my local machine:

             [exec] +1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     +1 tests included.  The patch appears to include 5 new or modified tests.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        
        Show
        Hong Tang added a comment - test-patch passes on my local machine: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 5 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Chris Douglas added a comment -

        Small nits:

        • Now that the *IndexMapper classes are in RandomAlgorithms, they can be removed from FilePool
        • In the usage, it could be clearer which parameters belong to the submission mode and job type
        • mapMaxSleepTime and reduceMaxSleepTime can be final

        Other than these, this looks good

        Show
        Chris Douglas added a comment - Small nits: Now that the *IndexMapper classes are in RandomAlgorithms , they can be removed from FilePool In the usage, it could be clearer which parameters belong to the submission mode and job type mapMaxSleepTime and reduceMaxSleepTime can be final Other than these, this looks good
        Hide
        Hong Tang added a comment -

        New patch that addresses Chris's comments. The usage output now looks like the following:

        Usage: gridmix [-generate <MiB>] [-users URI] [-Dname=value ...] <iopath> <trace>
          e.g. gridmix -generate 100m foo -
        Configuration parameters:
           General parameters:
               gridmix.output.directory                         : Output directory
               gridmix.client.submit.threads                    : Submitting threads
               gridmix.client.pending.queue.depth               : Queued job desc
               gridmix.user.resolve.class                       : User resolution class
               gridmix.job.type                                 : Job types (LOADJOB|SLEEPJOB)
           Parameters related to job submission:
               gridmix.job-submission.default-queue             : Default queue
               gridmix.job-submission.use-queue-in-trace        : Enable/disable using queues in trace
               gridmix.job-submission.policy                    : Job submission policy (REPLAY|STRESS|SERIAL)
           Parameters specific for LOADJOB:
               gridmix.key.fraction                             : Key fraction of rec
           Parameters specific for SLEEPJOB:
               gridmix.sleep.maptask-only                       : Whether to ignore reduce tasks
               gridmix.sleep.fake-locations                     : Number of fake locations for map tasks
               gridmix.sleep.max-map-time                       : Maximum map task runtime in mili-sec
               gridmix.sleep.max-reduce-time                    : Maximum reduce task runtime in mili-sec (merge+reduce)
           Parameters specific for STRESS submission throttling policy:
               gridmix.throttle.jobs-to-tracker-ratio           : jobs vs task-tracker ratio
               gridmix.throttle.maps.task-to-slot-ratio         : maps vs map-slot ratio
               gridmix.throttle.reduces.task-to-slot-ratio      : reduces vs reduce-slot ratio
               gridmix.throttle.maps.max-slot-share-per-job     : map-slot share per job
               gridmix.throttle.reducess.max-slot-share-per-job : reduce-slot share per job
        
        Show
        Hong Tang added a comment - New patch that addresses Chris's comments. The usage output now looks like the following: Usage: gridmix [-generate <MiB>] [-users URI] [-Dname=value ...] <iopath> <trace> e.g. gridmix -generate 100m foo - Configuration parameters: General parameters: gridmix.output.directory : Output directory gridmix.client.submit.threads : Submitting threads gridmix.client.pending.queue.depth : Queued job desc gridmix.user.resolve.class : User resolution class gridmix.job.type : Job types (LOADJOB|SLEEPJOB) Parameters related to job submission: gridmix.job-submission.default-queue : Default queue gridmix.job-submission.use-queue-in-trace : Enable/disable using queues in trace gridmix.job-submission.policy : Job submission policy (REPLAY|STRESS|SERIAL) Parameters specific for LOADJOB: gridmix.key.fraction : Key fraction of rec Parameters specific for SLEEPJOB: gridmix.sleep.maptask-only : Whether to ignore reduce tasks gridmix.sleep.fake-locations : Number of fake locations for map tasks gridmix.sleep.max-map-time : Maximum map task runtime in mili-sec gridmix.sleep.max-reduce-time : Maximum reduce task runtime in mili-sec (merge+reduce) Parameters specific for STRESS submission throttling policy: gridmix.throttle.jobs-to-tracker-ratio : jobs vs task-tracker ratio gridmix.throttle.maps.task-to-slot-ratio : maps vs map-slot ratio gridmix.throttle.reduces.task-to-slot-ratio : reduces vs reduce-slot ratio gridmix.throttle.maps.max-slot-share-per-job : map-slot share per job gridmix.throttle.reducess.max-slot-share-per-job : reduce-slot share per job
        Hide
        Chris Douglas added a comment -
             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 5 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        
        Show
        Chris Douglas added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 5 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Chris Douglas added a comment -

        +1

        I committed this. Thanks, Hong!

        Show
        Chris Douglas added a comment - +1 I committed this. Thanks, Hong!
        Hide
        Hong Tang added a comment -

        Delta patch incorporating changes reflecting Chris's comments.

        Show
        Hong Tang added a comment - Delta patch incorporating changes reflecting Chris's comments.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )

          People

          • Assignee:
            Hong Tang
            Reporter:
            Hong Tang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development