Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: contrib/gridmix
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Adds system tests for testing the compression emulation feature of Gridmix.
    • Tags:
      gridmix compression-emulation system-tests

      Description

      Develop system tests for the following cases.

      1. Enable a compression emulation and generated the data using gridmix and verify whether compressed input generated or not.
      2. Verify a Gridmix jobs map input, map output and reduce output compression ratios against the default compression ratios.
      3. Verify a Gridmix jobs map input, map output and reduce output compression ratios against user specified compression ratios.
      4. Verify a Gridmix jobs map input, map output compression ratios with file output compression format is false in original trace against default compression ratios.
      5. Verify a Gridmix jobs map input, map output compression ratios with file output compression format is false in original trace against user specified compression ratios.
      6. Verify a Gridmix jobs reduce output with file output compression format is true in original trace against the default compression ratios.
      7. Verify a Gridmix jobs reduce output with file output compression format is true in original trace against the user specified compression ratios.

      1. MAPREDUCE-2544-v2.patch
        211 kB
        Vinay Kumar Thota
      2. MAPREDUCE-2544.patch
        211 kB
        Vinay Kumar Thota

        Activity

        Vinay Kumar Thota created issue -
        Hide
        Vinay Kumar Thota added a comment -

        Please review the patch and let me know your comments.

        Show
        Vinay Kumar Thota added a comment - Please review the patch and let me know your comments.
        Vinay Kumar Thota made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-2544.patch [ 12480898 ]
        Hide
        Vinay Kumar Thota added a comment -

        Addressed Amar's offline comments.

        Show
        Vinay Kumar Thota added a comment - Addressed Amar's offline comments.
        Vinay Kumar Thota made changes -
        Attachment MAPREDUCE-2544-v2.patch [ 12480915 ]
        Hide
        Konstantin Boudnik added a comment -

        Hey Vinay. These aren't Herriot system tests, are they?

        Show
        Konstantin Boudnik added a comment - Hey Vinay. These aren't Herriot system tests, are they?
        Hide
        Amar Kamat added a comment -

        The latest patch looks good to me. +1.

        Show
        Amar Kamat added a comment - The latest patch looks good to me. +1.
        Hide
        Vinay Kumar Thota added a comment -

        Hey Vinay. These aren't Herriot system tests, are they?

        Yes, these are Herriot system tests and you can invoke them using a target test-system.

        Show
        Vinay Kumar Thota added a comment - Hey Vinay. These aren't Herriot system tests, are they? Yes, these are Herriot system tests and you can invoke them using a target test-system.
        Vinay Kumar Thota made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480915/MAPREDUCE-2544-v2.patch
        against trunk revision 1129771.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestMRCLI

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480915/MAPREDUCE-2544-v2.patch against trunk revision 1129771. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//console This message is automatically generated.
        Hide
        Vinay Kumar Thota added a comment -

        Findbugs and core tests issues are not related to this patch.

        Show
        Vinay Kumar Thota added a comment - Findbugs and core tests issues are not related to this patch.
        Hide
        Amar Kamat added a comment -

        I just committed the latest patch to trunk. Thanks Vinay!

        Show
        Amar Kamat added a comment - I just committed the latest patch to trunk. Thanks Vinay!
        Amar Kamat made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Release Note Adds system tests for testing the compression emulation feature of Gridmix.
        Resolution Fixed [ 1 ]
        Tags gridmix compression-emulation system-tests
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #707 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/707/)
        MAPREDUCE-2544. [Gridmix] Add compression emulation system tests to Gridmix. (Vinay Kumar Thota via amarrk)

        amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130147
        Files :

        • /hadoop/mapreduce/trunk/CHANGES.txt
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case1_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case2_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case3_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case4_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationForUncompressInAndCompressOut.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationForCompressInAndUncompressOut.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationEnableForAllTypesOfJobs.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixCompressionEmulationWithCompressInput.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixCompressedInputGeneration.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #707 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/707/ ) MAPREDUCE-2544 . [Gridmix] Add compression emulation system tests to Gridmix. (Vinay Kumar Thota via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130147 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case1_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case2_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case3_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case4_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationForUncompressInAndCompressOut.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationForCompressInAndUncompressOut.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationEnableForAllTypesOfJobs.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixCompressionEmulationWithCompressInput.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixCompressedInputGeneration.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Vinay Kumar Thota
            Reporter:
            Vinay Kumar Thota
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development