Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2544

Gridmix compression emulation system tests.

        Details

        • Type: Task Task
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 0.23.0
        • Component/s: contrib/gridmix
        • Labels:
          None
        • Hadoop Flags:
          Reviewed
        • Release Note:
          Adds system tests for testing the compression emulation feature of Gridmix.
        • Tags:
          gridmix compression-emulation system-tests

          Description

          Develop system tests for the following cases.

          1. Enable a compression emulation and generated the data using gridmix and verify whether compressed input generated or not.
          2. Verify a Gridmix jobs map input, map output and reduce output compression ratios against the default compression ratios.
          3. Verify a Gridmix jobs map input, map output and reduce output compression ratios against user specified compression ratios.
          4. Verify a Gridmix jobs map input, map output compression ratios with file output compression format is false in original trace against default compression ratios.
          5. Verify a Gridmix jobs map input, map output compression ratios with file output compression format is false in original trace against user specified compression ratios.
          6. Verify a Gridmix jobs reduce output with file output compression format is true in original trace against the default compression ratios.
          7. Verify a Gridmix jobs reduce output with file output compression format is true in original trace against the user specified compression ratios.

          1. Text File
            MAPREDUCE-2544.patch
            211 kB
            Vinay Kumar Thota
          2. Text File
            MAPREDUCE-2544-v2.patch
            211 kB
            Vinay Kumar Thota

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Vinay Kumar Thota added a comment -

            Please review the patch and let me know your comments.

            Show
            Vinay Kumar Thota added a comment - Please review the patch and let me know your comments.
            Hide
            Permalink
            Vinay Kumar Thota added a comment -

            Addressed Amar's offline comments.

            Show
            Vinay Kumar Thota added a comment - Addressed Amar's offline comments.
            Hide
            Permalink
            Konstantin Boudnik added a comment -

            Hey Vinay. These aren't Herriot system tests, are they?

            Show
            Konstantin Boudnik added a comment - Hey Vinay. These aren't Herriot system tests, are they?
            Hide
            Permalink
            Amar Kamat added a comment -

            The latest patch looks good to me. +1.

            Show
            Amar Kamat added a comment - The latest patch looks good to me. +1.
            Hide
            Permalink
            Vinay Kumar Thota added a comment -

            Hey Vinay. These aren't Herriot system tests, are they?

            Yes, these are Herriot system tests and you can invoke them using a target test-system.

            Show
            Vinay Kumar Thota added a comment - Hey Vinay. These aren't Herriot system tests, are they? Yes, these are Herriot system tests and you can invoke them using a target test-system.
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12480915/MAPREDUCE-2544-v2.patch
            against trunk revision 1129771.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 24 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed these core unit tests:
            org.apache.hadoop.cli.TestMRCLI

            +1 contrib tests. The patch passed contrib unit tests.

            +1 system test framework. The patch passed system test framework compile.

            Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//testReport/
            Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480915/MAPREDUCE-2544-v2.patch against trunk revision 1129771. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/327//console This message is automatically generated.
            Hide
            Permalink
            Vinay Kumar Thota added a comment -

            Findbugs and core tests issues are not related to this patch.

            Show
            Vinay Kumar Thota added a comment - Findbugs and core tests issues are not related to this patch.
            Hide
            Permalink
            Amar Kamat added a comment -

            I just committed the latest patch to trunk. Thanks Vinay!

            Show
            Amar Kamat added a comment - I just committed the latest patch to trunk. Thanks Vinay!
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk-Commit #707 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/707/)
            MAPREDUCE-2544. [Gridmix] Add compression emulation system tests to Gridmix. (Vinay Kumar Thota via amarrk)

            amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130147
            Files :

            • /hadoop/mapreduce/trunk/CHANGES.txt
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case1_trace.json.gz
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case2_trace.json.gz
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case3_trace.json.gz
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case4_trace.json.gz
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationForUncompressInAndCompressOut.java
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationForCompressInAndUncompressOut.java
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationEnableForAllTypesOfJobs.java
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixCompressionEmulationWithCompressInput.java
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixCompressedInputGeneration.java
            • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java
            Show
            Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #707 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/707/ ) MAPREDUCE-2544 . [Gridmix] Add compression emulation system tests to Gridmix. (Vinay Kumar Thota via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130147 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case1_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case2_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case3_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/compression_case4_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationForUncompressInAndCompressOut.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationForCompressInAndUncompressOut.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationEnableForAllTypesOfJobs.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixCompressionEmulationWithCompressInput.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixCompressedInputGeneration.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

            Show
            Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )

              People

              • Assignee:
                Vinay Kumar Thota
                Reporter:
                Vinay Kumar Thota
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development