Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: benchmarks, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Slive currently uses single reducer. It could use multiple ones.

      1. SliveMultiR.patch
        22 kB
        Konstantin Shvachko
      2. SliveMultiR.patch
        21 kB
        Konstantin Shvachko
      3. SliveMultiR.patch
        20 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment -

          Slive maps output different stats of the operations performed by the test. The stats are currently aggregated by a single reducer, which may run for a long time if the amount of generated data is large.
          I propose to add a parameter to Slive args, which specifies the number of reducers R. Then SlivePartitioner for each output stat calculates a hash value of the operation type, modular R. This defines the reducer number.

          Show
          Konstantin Shvachko added a comment - Slive maps output different stats of the operations performed by the test. The stats are currently aggregated by a single reducer, which may run for a long time if the amount of generated data is large. I propose to add a parameter to Slive args, which specifies the number of reducers R. Then SlivePartitioner for each output stat calculates a hash value of the operation type, modular R. This defines the reducer number.
          Hide
          Konstantin Shvachko added a comment -

          In the patch:

          • Introduced SlivePartitioner, and changed code to collect reports from multiple output files.
          • Made Slive a Tool, so that people could specify generic options.
          • Updated TestSlive to run with 2 mappers and 2 reducers. Also tuned it up to run in eclipse, and fixed some stream closing issue.
          • Improved some JavaDoc and log messages.
          • updated the design document in
            https://issues.apache.org/jira/secure/attachment/12448004/SLiveTest.pdf
          Show
          Konstantin Shvachko added a comment - In the patch: Introduced SlivePartitioner, and changed code to collect reports from multiple output files. Made Slive a Tool, so that people could specify generic options. Updated TestSlive to run with 2 mappers and 2 reducers. Also tuned it up to run in eclipse, and fixed some stream closing issue. Improved some JavaDoc and log messages. updated the design document in https://issues.apache.org/jira/secure/attachment/12448004/SLiveTest.pdf
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12448005/SliveMultiR.patch
          against trunk revision 957772.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 31 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448005/SliveMultiR.patch against trunk revision 957772. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 31 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          Fixed the audit warning.
          I am going to commit it if there are no objections. Please feel free to review this.

          Show
          Konstantin Shvachko added a comment - Fixed the audit warning. I am going to commit it if there are no objections. Please feel free to review this.
          Hide
          Konstantin Shvachko added a comment -

          The patch adds Slive to MapredTestDriver. Ravi asked for this as it simplifies test launching.

          Show
          Konstantin Shvachko added a comment - The patch adds Slive to MapredTestDriver. Ravi asked for this as it simplifies test launching.
          Hide
          Ravi Phulari added a comment -

          Thanks for adding Slive to Driver Konstantin.
          Patch looks good to me.

          Show
          Ravi Phulari added a comment - Thanks for adding Slive to Driver Konstantin. Patch looks good to me.
          Hide
          Ravi Phulari added a comment -

          Sorry, I missed one thing earlier .
          One correction is needed in MapredTestDriver.java
          We need to make SliveTest visible through TestDriver which can be done by adding "SliveTest" as first parameter to pgd.addClass()

          pgd.addClass(SliveTest.class.getSimpleName(), SliveTest.class, 
                    "HDFS Stress Test and Live Data Verification.");
          

          Should be like -

          pgd.addClass("SliveTest",SliveTest.class.getSimpleName(), SliveTest.class, 
                    "HDFS Stress Test and Live Data Verification.");
          
          Show
          Ravi Phulari added a comment - Sorry, I missed one thing earlier . One correction is needed in MapredTestDriver.java We need to make SliveTest visible through TestDriver which can be done by adding "SliveTest" as first parameter to pgd.addClass() pgd.addClass(SliveTest.class.getSimpleName(), SliveTest.class, "HDFS Stress Test and Live Data Verification."); Should be like - pgd.addClass("SliveTest",SliveTest.class.getSimpleName(), SliveTest.class, "HDFS Stress Test and Live Data Verification.");
          Hide
          Konstantin Shvachko added a comment -

          Ravi, SliveTest.class.getSimpleName() returns string "SliveTest". Should work fine as is. Could you pls verify again.

          Show
          Konstantin Shvachko added a comment - Ravi, SliveTest.class.getSimpleName() returns string "SliveTest" . Should work fine as is. Could you pls verify again.
          Hide
          Ravi Phulari added a comment -

          I have verified , it works. Sorry for the delay Konstantin.

          Show
          Ravi Phulari added a comment - I have verified , it works. Sorry for the delay Konstantin.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.

          Show
          Konstantin Shvachko added a comment - I just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development