Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2554

Gridmix distributed cache emulation system tests.

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: contrib/gridmix
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Adds distributed cache related system tests to Gridmix.

      Description

      1.Verify the emulation of HDFS and Local FS distributed cache files against the given input trace file.
      2.Verify the Gridmix emulation of HDFS distributed cache files of different visibilities.
      3.Verify the Gridmix emulation of HDFS distributed cache file which uses different jobs that are submitted with different users.
      4.Verify the emulation of local FS distributed cache files.
      5.Verify the Gridmix emulation of HDFS private distributed cache file.
      6.Verify the Gridmix emulation of HDFS public distributed cache file.
      7.Verify the Gridmix emulation of Multiple HDFS private distributed cache files.
      8.Verify the Gridmix emulation of Multiple HDFS public distributed cache files.

      1. MAPREDUCE-2554.patch
        603 kB
        Vinay Kumar Thota
      2. MAPREDUCE-2554-v2.patch
        605 kB
        Vinay Kumar Thota

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #713 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/713/)
        MAPREDUCE-2554. [Gridmix] Add distributed cache emulation system tests to Gridmix. (Vinay Kumar Thota via amarrk)

        amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1131515
        Files :

        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case7_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case8_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case9_trace.json.gz
        • /hadoop/mapreduce/trunk/CHANGES.txt
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixEmulationOfMultipleHDFSPrivateDCFiles.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixEmulationOfHDFSPublicDCFile.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestEmulationOfLocalFSDCFiles.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/UtilsForGridmix.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestEmulationOfHDFSAndLocalFSDCFiles.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestEmulationOfHDFSDCFileUsesMultipleJobs.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestEmulationOfHDFSDCFilesWithDifferentVisibilities.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixEmulationOfMultipleHDFSPublicDCFiles.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixEmulationOfHDFSPrivateDCFile.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case1_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case2_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case3_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/GridmixSystemTestCase.java
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case4_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case5_trace.json.gz
        • /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case6_trace.json.gz
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #713 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/713/ ) MAPREDUCE-2554 . [Gridmix] Add distributed cache emulation system tests to Gridmix. (Vinay Kumar Thota via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1131515 Files : /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case7_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case8_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case9_trace.json.gz /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixEmulationOfMultipleHDFSPrivateDCFiles.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixEmulationOfHDFSPublicDCFile.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestEmulationOfLocalFSDCFiles.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/UtilsForGridmix.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestEmulationOfHDFSAndLocalFSDCFiles.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestEmulationOfHDFSDCFileUsesMultipleJobs.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestEmulationOfHDFSDCFilesWithDifferentVisibilities.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixEmulationOfMultipleHDFSPublicDCFiles.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestGridmixEmulationOfHDFSPrivateDCFile.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case1_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case2_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case3_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/GridmixSystemTestCase.java /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case4_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case5_trace.json.gz /hadoop/mapreduce/trunk/src/contrib/gridmix/src/test/system/resources/distcache_case6_trace.json.gz
        Hide
        Amar Kamat added a comment -

        I just committed the latest patch to trunk on Ravi's behalf. Thanks Vinay and Ravi!

        Show
        Amar Kamat added a comment - I just committed the latest patch to trunk on Ravi's behalf. Thanks Vinay and Ravi!
        Hide
        Vinay Kumar Thota added a comment -

        Findbugs are not related to this patch.Also TestMRCLI failure is not related this patch.

        Show
        Vinay Kumar Thota added a comment - Findbugs are not related to this patch.Also TestMRCLI failure is not related this patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12481316/MAPREDUCE-2554-v2.patch
        against trunk revision 1130994.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 46 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestMRCLI

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/341//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/341//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/341//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481316/MAPREDUCE-2554-v2.patch against trunk revision 1130994. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 46 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/341//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/341//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/341//console This message is automatically generated.
        Hide
        Ravi Gummadi added a comment -

        Patch looks fine to me.
        +1

        Show
        Ravi Gummadi added a comment - Patch looks fine to me. +1
        Hide
        Vinay Kumar Thota added a comment -

        Uploaded another new patch with minor changes.Ran the system tests with the patch and no issues were found.

        Show
        Vinay Kumar Thota added a comment - Uploaded another new patch with minor changes.Ran the system tests with the patch and no issues were found.
        Hide
        Vinay Kumar Thota added a comment -

        Please review the patch and let me know the comments.

        Show
        Vinay Kumar Thota added a comment - Please review the patch and let me know the comments.

          People

          • Assignee:
            Yiting Wu
            Reporter:
            Vinay Kumar Thota
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development