Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: benchmarks, contrib/gridmix
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Gridmix now prints a summary information after every run. It summarizes the runs w.r.t input trace details, input data statistics, cli arguments, data-gen runtime, simulation runtimes etc and also the cluster w.r.t map slots, reduce slots, jobtracker-address, hdfs-address etc.
      Show
      Gridmix now prints a summary information after every run. It summarizes the runs w.r.t input trace details, input data statistics, cli arguments, data-gen runtime, simulation runtimes etc and also the cluster w.r.t map slots, reduce slots, jobtracker-address, hdfs-address etc.
    • Tags:
      gridmix report summary

      Description

      Gridmix doesn't warn the user if any of the jobs in the mix fail... it probably should printout a summary of the jobs and other statistics at the end too.

      1. gridmix-summary-v1.3.patch
        45 kB
        Amar Kamat
      2. gridmix-summary-v1.4.patch
        45 kB
        Amar Kamat
      3. gridmix-summary-v1.5.patch
        45 kB
        Amar Kamat
      4. gridmix-summary-v1.6.patch
        45 kB
        Amar Kamat
      5. gridmix-summary-v1.7.patch
        45 kB
        Amar Kamat
      6. gridmix-summary-v1.8.patch
        45 kB
        Amar Kamat

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #732 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/732/)
        MAPREDUCE-2596. [Gridmix] Summarize Gridmix runs. (amarrk)

        amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144403
        Files :

        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/CompressionEmulationUtil.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Summarizer.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GenerateData.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobMonitor.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobSubmitter.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/ClusterSummarizer.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/ExecutionSummarizer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #732 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/732/ ) MAPREDUCE-2596 . [Gridmix] Summarize Gridmix runs. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144403 Files : /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/CompressionEmulationUtil.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Summarizer.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GenerateData.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobMonitor.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobSubmitter.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/ClusterSummarizer.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/ExecutionSummarizer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #736 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/736/)
        MAPREDUCE-2596. [Gridmix] Summarize Gridmix runs. (amarrk)

        amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144403
        Files :

        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/CompressionEmulationUtil.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Summarizer.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GenerateData.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobMonitor.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobSubmitter.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/ClusterSummarizer.java
        • /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/ExecutionSummarizer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #736 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/736/ ) MAPREDUCE-2596 . [Gridmix] Summarize Gridmix runs. (amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144403 Files : /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/CompressionEmulationUtil.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Summarizer.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/GenerateData.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobMonitor.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobSubmitter.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/ClusterSummarizer.java /hadoop/common/trunk/mapreduce/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/ExecutionSummarizer.java
        Amar Kamat made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Release Note Gridmix now prints a summary information after every run. It summarizes the runs w.r.t input trace details, input data statistics, cli arguments, data-gen runtime, simulation runtimes etc and also the cluster w.r.t map slots, reduce slots, jobtracker-address, hdfs-address etc.
        Resolution Fixed [ 1 ]
        Hide
        Amar Kamat added a comment -

        I just committed the latest patch to trunk. Thanks Ravi and Vinay for reviewing this.

        Show
        Amar Kamat added a comment - I just committed the latest patch to trunk. Thanks Ravi and Vinay for reviewing this.
        Hide
        Amar Kamat added a comment -

        None of the failed tests seem related to this patch. Failing tests are mostly RAID tests and CLI tests.

        Show
        Amar Kamat added a comment - None of the failed tests seem related to this patch. Failing tests are mostly RAID tests and CLI tests.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12485686/gridmix-summary-v1.8.patch
        against trunk revision 1144097.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestMRCLI
        org.apache.hadoop.fs.TestFileSystem

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/445//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/445//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/445//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485686/gridmix-summary-v1.8.patch against trunk revision 1144097. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/445//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/445//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/445//console This message is automatically generated.
        Hide
        Ravi Gummadi added a comment -

        Patch looks fine to me.
        +1

        Show
        Ravi Gummadi added a comment - Patch looks fine to me. +1
        Amar Kamat made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Tags gridmix report summary
        Amar Kamat made changes -
        Attachment gridmix-summary-v1.8.patch [ 12485686 ]
        Hide
        Amar Kamat added a comment -

        Attaching a final version of the patch for commit. Includes minor changes w.r.t variable/api naming.

        test-patch passed on my box.

        Show
        Amar Kamat added a comment - Attaching a final version of the patch for commit. Includes minor changes w.r.t variable/api naming. test-patch passed on my box.
        Amar Kamat made changes -
        Attachment gridmix-summary-v1.7.patch [ 12485532 ]
        Hide
        Amar Kamat added a comment -

        Attaching a new patch with some minor offline comments from Ravi and Vinay. test-patch passed. The changes are in code comments and log statements.

        Show
        Amar Kamat added a comment - Attaching a new patch with some minor offline comments from Ravi and Vinay. test-patch passed. The changes are in code comments and log statements.
        Amar Kamat made changes -
        Attachment gridmix-summary-v1.6.patch [ 12484626 ]
        Hide
        Amar Kamat added a comment -

        Attaching a new patch that fixes a corner case (only data gen). test-patch passed on my box.

        Show
        Amar Kamat added a comment - Attaching a new patch that fixes a corner case (only data gen). test-patch passed on my box.
        Amar Kamat made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Amar Kamat added a comment -

        The failed testcases are
        1. org.apache.hadoop.mapred.TestNodeRefresh.testMRRefreshDecommissioning
        2. org.apache.hadoop.raid.TestRaidNode.testPathFilter
        3. org.apache.hadoop.fs.TestFileSystem.testCommandFormat
        4. org.apache.hadoop.cli.TestMRCLI.testAll

        None of them are related to this patch.

        Show
        Amar Kamat added a comment - The failed testcases are 1. org.apache.hadoop.mapred.TestNodeRefresh.testMRRefreshDecommissioning 2. org.apache.hadoop.raid.TestRaidNode.testPathFilter 3. org.apache.hadoop.fs.TestFileSystem.testCommandFormat 4. org.apache.hadoop.cli.TestMRCLI.testAll None of them are related to this patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12484410/gridmix-summary-v1.5.patch
        against trunk revision 1139400.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestMRCLI
        org.apache.hadoop.fs.TestFileSystem
        org.apache.hadoop.mapred.TestNodeRefresh

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/430//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/430//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/430//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12484410/gridmix-summary-v1.5.patch against trunk revision 1139400. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.mapred.TestNodeRefresh -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/430//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/430//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/430//console This message is automatically generated.
        Amar Kamat made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amar Kamat made changes -
        Attachment gridmix-summary-v1.5.patch [ 12484410 ]
        Hide
        Amar Kamat added a comment -

        Attaching a new patch that removes the distinction between jobs that failed before submission and jobs that failed after job submission. Also fixed a bug in JobSubmitter and JobMonitor.

        test-patch and gridmix junit tests passed in my box.

        Show
        Amar Kamat added a comment - Attaching a new patch that removes the distinction between jobs that failed before submission and jobs that failed after job submission. Also fixed a bug in JobSubmitter and JobMonitor. test-patch and gridmix junit tests passed in my box.
        Hide
        Vinay Kumar Thota added a comment -

        Patch looks good to me.
        +1.

        Show
        Vinay Kumar Thota added a comment - Patch looks good to me. +1.
        Amar Kamat made changes -
        Attachment gridmix-summary-v1.4.patch [ 12483927 ]
        Hide
        Amar Kamat added a comment -

        Attaching a new patch incorporating Vinay's review comments. test-patch passed on my box.

        Show
        Amar Kamat added a comment - Attaching a new patch incorporating Vinay's review comments. test-patch passed on my box.
        Hide
        Amar Kamat added a comment -

        Thanks for the review Vinay.

        You can define the above method as static method right,because cluster information is same for across the jobs in a gridmix run right.

        I cannot define it as static as the member variables of that class are being set in the method.

        I think, no newline at end of file will cause an issues while adding the patch.Make sure to give a space at end of the file.

        The patch cleanly applies and test-patch passed. I have verified this.

        The above instructions will change to if(condition){}elseif(conditon){}else{} format right.

        +1.

        Add java doc comments at least for public methods.

        I have intentionally not added comments for overridden apis. I will check again and add the missing ones.

        Show
        Amar Kamat added a comment - Thanks for the review Vinay. You can define the above method as static method right,because cluster information is same for across the jobs in a gridmix run right. I cannot define it as static as the member variables of that class are being set in the method. I think, no newline at end of file will cause an issues while adding the patch.Make sure to give a space at end of the file. The patch cleanly applies and test-patch passed. I have verified this. The above instructions will change to if(condition){}elseif(conditon){}else{} format right. +1. Add java doc comments at least for public methods. I have intentionally not added comments for overridden apis. I will check again and add the missing ones.
        Hide
        Vinay Kumar Thota added a comment -

        I have reviewed the patch and few minor comments.

         
        +  @SuppressWarnings("deprecation")
        +  public void update(ClusterStats item) {
        +    try {
        +      numBlacklistedTrackers = item.getStatus().getBlacklistedTrackers();
        +      numActiveTrackers = item.getStatus().getTaskTrackers();
        +      maxMapTasks = item.getStatus().getMaxMapTasks();
        +      maxReduceTasks = item.getStatus().getMaxReduceTasks();
        +    } catch (Exception e) {
        +      long time = System.currentTimeMillis();
        +      LOG.info("Error in processing cluster status at " 
        +               + FastDateFormat.getInstance().format(time));
        +    }
        +  }
        

        You can define the above method as static method right,because cluster information is same for across the jobs in a gridmix run right.

        \ No newline at end of file

        I think, no newline at end of file will cause an issues while adding the patch.Make sure to give a space at end of the file.

         
        +    if (job.isSuccessful()) {
        +      ++totalSuccessfulJobs;
        +    } else {
        +      if (job.isComplete()) {
        +        ++totalFailedJobs;
        +      } else {
        +        //TODO Check
        +        ++totalSubmitFailedJobs;
        +      }
        +    }
        

        The above instructions will change to if(condition){}elseif(conditon){}else{} format right.

        Add java doc comments at least for public methods.

        Show
        Vinay Kumar Thota added a comment - I have reviewed the patch and few minor comments. + @SuppressWarnings("deprecation") + public void update(ClusterStats item) { + try { + numBlacklistedTrackers = item.getStatus().getBlacklistedTrackers(); + numActiveTrackers = item.getStatus().getTaskTrackers(); + maxMapTasks = item.getStatus().getMaxMapTasks(); + maxReduceTasks = item.getStatus().getMaxReduceTasks(); + } catch (Exception e) { + long time = System.currentTimeMillis(); + LOG.info("Error in processing cluster status at " + + FastDateFormat.getInstance().format(time)); + } + } You can define the above method as static method right,because cluster information is same for across the jobs in a gridmix run right. \ No newline at end of file I think, no newline at end of file will cause an issues while adding the patch.Make sure to give a space at end of the file. + if (job.isSuccessful()) { + ++totalSuccessfulJobs; + } else { + if (job.isComplete()) { + ++totalFailedJobs; + } else { + //TODO Check + ++totalSubmitFailedJobs; + } + } The above instructions will change to if(condition){}elseif(conditon){}else{} format right. Add java doc comments at least for public methods.
        Amar Kamat made changes -
        Attachment gridmix-summary-v1.3.patch [ 12483551 ]
        Amar Kamat made changes -
        Attachment gridmix-memory-emulation-v1.3.patch [ 12483550 ]
        Amar Kamat made changes -
        Attachment gridmix-memory-emulation-v1.3.patch [ 12483550 ]
        Hide
        Amar Kamat added a comment -

        Attaching a patch that summarizes the gridmix run. This is how the summary looks like:

        Execution Summary:-
        Input trace: file:/home/user/hadoop/mapreduce/job_201106081604_0006.json
        Input trace signature: 621979b3d484a2d824bd5d9a721dc085
        Total number of jobs in trace: 1
        Expected input data size: N/A
        Input data statistics: Input data size: 5.0m, Number of files: 1, Compression: true
        Total number of jobs processed: 1
        Total number of successful jobs: 1
        Total number of failed jobs: 0
        Total number of jobs failed after submission: 0
        Total number of jobs failed before submission: 0
        Total number of map tasks launched: 1
        Total number of reduce task launched: 1
        Gridmix start time: 6/23/11 12:37 PM
        Gridmix end time: 6/23/11 12:40 PM
        Gridmix simulation start time: 6/23/11 12:37 PM
        Gridmix runtime: 2mins, 57sec
        Time spent in initialization (data-gen etc): 2sec
        Time spent in simulation: 2mins, 54sec
        Gridmix configuration parameters: -Dgridmix.output.directory=output -Dmapreduce.job.ubertask.enable=false -Dgridmix.min.file.size=1024 -Dgridmix.compression-emulation.enable=true input file:///home/user/hadoop/mapreduce/job_201106081604_0006.json
        Gridmix job submission policy: STRESS
        Gridmix resolver: org.apache.hadoop.mapred.gridmix.SubmitterUserResolver
        
        Cluster Summary:-
        JobTracker: localhost:9520
        FileSystem: localhost:9500
        Number of blacklisted trackers: 0
        Number of active trackers: 1
        Max map task capacity: 2
        Max reduce task capacity: 2
        

        test-patch and gridmix junit tests passed.

        Todo:
        1. job level summary (current-jobid, original-jobid, runtime characteristics, input output characteristics etc)
        2. ability to record the summary in a specified file and in a specified format (xml/json/csv)
        3. Test this on a slightly larger cluster.

        The first cut is intentionally kept simple.

        Show
        Amar Kamat added a comment - Attaching a patch that summarizes the gridmix run. This is how the summary looks like: Execution Summary:- Input trace: file:/home/user/hadoop/mapreduce/job_201106081604_0006.json Input trace signature: 621979b3d484a2d824bd5d9a721dc085 Total number of jobs in trace: 1 Expected input data size: N/A Input data statistics: Input data size: 5.0m, Number of files: 1, Compression: true Total number of jobs processed: 1 Total number of successful jobs: 1 Total number of failed jobs: 0 Total number of jobs failed after submission: 0 Total number of jobs failed before submission: 0 Total number of map tasks launched: 1 Total number of reduce task launched: 1 Gridmix start time: 6/23/11 12:37 PM Gridmix end time: 6/23/11 12:40 PM Gridmix simulation start time: 6/23/11 12:37 PM Gridmix runtime: 2mins, 57sec Time spent in initialization (data-gen etc): 2sec Time spent in simulation: 2mins, 54sec Gridmix configuration parameters: -Dgridmix.output.directory=output -Dmapreduce.job.ubertask.enable=false -Dgridmix.min.file.size=1024 -Dgridmix.compression-emulation.enable=true input file:///home/user/hadoop/mapreduce/job_201106081604_0006.json Gridmix job submission policy: STRESS Gridmix resolver: org.apache.hadoop.mapred.gridmix.SubmitterUserResolver Cluster Summary:- JobTracker: localhost:9520 FileSystem: localhost:9500 Number of blacklisted trackers: 0 Number of active trackers: 1 Max map task capacity: 2 Max reduce task capacity: 2 test-patch and gridmix junit tests passed. Todo: 1. job level summary (current-jobid, original-jobid, runtime characteristics, input output characteristics etc) 2. ability to record the summary in a specified file and in a specified format (xml/json/csv) 3. Test this on a slightly larger cluster. The first cut is intentionally kept simple.
        Amar Kamat made changes -
        Assignee Amar Kamat [ amar_kamat ]
        Amar Kamat made changes -
        Field Original Value New Value
        Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
        Key HADOOP-4338 MAPREDUCE-2596
        Issue Type Bug [ 1 ] Improvement [ 4 ]
        Affects Version/s 0.23.0 [ 12315570 ]
        Affects Version/s 0.20.0 [ 12313438 ]
        Component/s benchmarks [ 12312985 ]
        Component/s contrib/gridmix [ 12313086 ]
        Component/s benchmarks [ 12312428 ]
        Arun C Murthy created issue -

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Arun C Murthy
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development