Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4106 Fix skipping tests in mapreduce
  3. MAPREDUCE-4110

Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath & org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: mrv2, test
    • Labels:
      None

      Description

      Sub Project : hadoop-mapreduce-client-jobclient

      Running org.apache.hadoop.mapred.TestMiniMRClasspath
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.032 sec
      
      Running org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
      

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1042 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1042/)
        MAPREDUCE-4110. Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1042 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1042/ ) MAPREDUCE-4110 . Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1007 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1007/)
        MAPREDUCE-4110. Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1007 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1007/ ) MAPREDUCE-4110 . Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2034 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2034/)
        MAPREDUCE-4110. Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546)

        Result = ABORTED
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2034 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2034/ ) MAPREDUCE-4110 . Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2097 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2097/)
        MAPREDUCE-4110. Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2097 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2097/ ) MAPREDUCE-4110 . Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2022 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2022/)
        MAPREDUCE-4110. Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2022 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2022/ ) MAPREDUCE-4110 . Fix tests in TestMiniMRClasspath & TestMiniMRWithDFSWithDistinctUsers (Devaraj K via bobby) (Revision 1310546) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310546 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClasspath.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRWithDFSWithDistinctUsers.java
        Hide
        Robert Joseph Evans added a comment -

        Thanks Devaraj,

        I put this into branch-2, and trunk

        Show
        Robert Joseph Evans added a comment - Thanks Devaraj, I put this into branch-2, and trunk
        Hide
        Robert Joseph Evans added a comment -

        I know that MiniMRCluster internally starts the MiniMRYarnCluser, but why do we need to have an extra layer in here. The MiniMRCluser shim was added so that projects like Oozie and Pig can have a single test that will work for either Yarn or MRV1. Internally our tests are never going to go back to be run on MRV1 so why not update them to use the new code?

        But looking at it more now I see that it is actually extending the scope of this JIRA, perhaps a bit too much. I filed MAPREDUCE-4118 so we can make these changes.

        +1 for the original patch.

        Show
        Robert Joseph Evans added a comment - I know that MiniMRCluster internally starts the MiniMRYarnCluser, but why do we need to have an extra layer in here. The MiniMRCluser shim was added so that projects like Oozie and Pig can have a single test that will work for either Yarn or MRV1. Internally our tests are never going to go back to be run on MRV1 so why not update them to use the new code? But looking at it more now I see that it is actually extending the scope of this JIRA, perhaps a bit too much. I filed MAPREDUCE-4118 so we can make these changes. +1 for the original patch.
        Hide
        Devaraj K added a comment -

        Thanks a lot Bobby for looking into the patch.

        However, I would prefer to see the tests updated to use the MiniMRYarnCluster instead of the MiniMRCluster, as these are our own internal tests. If you would prefer to file a separate JIRA for that I am fine with it.

        MiniMRCluster internally starts the MiniMRYarnCluster and uses it. I don't think it is required to change in the test cases explicitly.

        I am also not really sure that we need TestMiniMRClasspath.testClassPath. There are already numerous tests that verify that we can run a word count job while setting the classpath. Adding in another, even though it predates some of the ones currently used, just seems like it would slow down testing and not improve the overall quality of the unit tests.

        IMO, better we should have dedicated test for this instead of depending on other tests to verify. If you feel not really required, please let me know I will remove it and update the patch.

        Show
        Devaraj K added a comment - Thanks a lot Bobby for looking into the patch. However, I would prefer to see the tests updated to use the MiniMRYarnCluster instead of the MiniMRCluster, as these are our own internal tests. If you would prefer to file a separate JIRA for that I am fine with it. MiniMRCluster internally starts the MiniMRYarnCluster and uses it. I don't think it is required to change in the test cases explicitly. I am also not really sure that we need TestMiniMRClasspath.testClassPath. There are already numerous tests that verify that we can run a word count job while setting the classpath. Adding in another, even though it predates some of the ones currently used, just seems like it would slow down testing and not improve the overall quality of the unit tests. IMO, better we should have dedicated test for this instead of depending on other tests to verify. If you feel not really required, please let me know I will remove it and update the patch.
        Hide
        Robert Joseph Evans added a comment -

        The changes look good for the most part.

        However, I would prefer to see the tests updated to use the MiniMRYarnCluster instead of the MiniMRCluster, as these are our own internal tests. If you would prefer to file a separate JIRA for that I am fine with it.

        I am also not really sure that we need TestMiniMRClasspath.testClassPath. There are already numerous tests that verify that we can run a word count job while setting the classpath. Adding in another, even though it predates some of the ones currently used, just seems like it would slow down testing and not improve the overall quality of the unit tests.

        Show
        Robert Joseph Evans added a comment - The changes look good for the most part. However, I would prefer to see the tests updated to use the MiniMRYarnCluster instead of the MiniMRCluster, as these are our own internal tests. If you would prefer to file a separate JIRA for that I am fine with it. I am also not really sure that we need TestMiniMRClasspath.testClassPath. There are already numerous tests that verify that we can run a word count job while setting the classpath. Adding in another, even though it predates some of the ones currently used, just seems like it would slow down testing and not improve the overall quality of the unit tests.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521481/MAPREDUCE-4110.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 14 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2160//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2160//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521481/MAPREDUCE-4110.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2160//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2160//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        Attached patch to fix this.

        Show
        Devaraj K added a comment - Attached patch to fix this.

          People

          • Assignee:
            Devaraj K
            Reporter:
            Devaraj K
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development