Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4642

MiniMRClientClusterFactory should not use job.setJar()

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, MiniMRClientClusterFactory does job.setJar(callerJar) so that the callerJar is added to the cache in MR2. However, this makes the resulting configuration inconsistent between MR1 and MR2 as in MR1 the job jar is not set and in MR2 its set to the callerJar. This difference can also cause some tests to fail in Oozie. We should instead use the job.addCacheFile() method.

      1. MAPREDUCE-4642.patch
        1.0 kB
        Robert Kanter
      2. MAPREDUCE-4642.patch
        1 kB
        Robert Kanter

        Activity

        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544133/MAPREDUCE-4642.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2826//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2826//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544133/MAPREDUCE-4642.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2826//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2826//console This message is automatically generated.
        Hide
        Robert Kanter added a comment -

        New patch uploads the callerJar to HDFS instead of using file://

        Show
        Robert Kanter added a comment - New patch uploads the callerJar to HDFS instead of using file://
        Hide
        Robert Kanter added a comment -

        and is more similar to how the MRAppJar is handled.

        Show
        Robert Kanter added a comment - and is more similar to how the MRAppJar is handled.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544170/MAPREDUCE-4642.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2829//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2829//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544170/MAPREDUCE-4642.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2829//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2829//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        +1

        Show
        Alejandro Abdelnur added a comment - +1
        Hide
        Alejandro Abdelnur added a comment -

        Thanks Robert. Committed to trunk and branch-2.

        Show
        Alejandro Abdelnur added a comment - Thanks Robert. Committed to trunk and branch-2.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2707 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2707/)
        MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2707 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2707/ ) MAPREDUCE-4642 . MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2770 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2770/)
        MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2770 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2770/ ) MAPREDUCE-4642 . MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2731 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2731/)
        MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2731 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2731/ ) MAPREDUCE-4642 . MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1159/)
        MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1159/ ) MAPREDUCE-4642 . MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1190/)
        MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1190/ ) MAPREDUCE-4642 . MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) (Revision 1382093) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
        Hide
        Arun C Murthy added a comment -

        Sorry to come in late.

        I'm confused - setJar is a standard user-api - this seems like a bug in Oozie if it can't deal with it?

        In the worst case - we can fix MR1 to use setJar too, but this fix seems off?

        Show
        Arun C Murthy added a comment - Sorry to come in late. I'm confused - setJar is a standard user-api - this seems like a bug in Oozie if it can't deal with it? In the worst case - we can fix MR1 to use setJar too, but this fix seems off?
        Hide
        Alejandro Abdelnur added a comment -

        Arun, MR1 is just fine. The problem was in the MiniMRClientClusterFactory.java (part of the adapter in YARN to mimic MR1 API) which was using the setJar() to set up minicluster classes in the classpath. Thus if the job running against the minicluster was trying to use setJar(), it would not work. With this fix avoid using setJar(), thus allowing testcases submitting jobs using setJar() to work as expected. Hope this clarifies.

        Show
        Alejandro Abdelnur added a comment - Arun, MR1 is just fine. The problem was in the MiniMRClientClusterFactory.java (part of the adapter in YARN to mimic MR1 API) which was using the setJar() to set up minicluster classes in the classpath. Thus if the job running against the minicluster was trying to use setJar(), it would not work. With this fix avoid using setJar(), thus allowing testcases submitting jobs using setJar() to work as expected. Hope this clarifies.

          People

          • Assignee:
            Robert Kanter
            Reporter:
            Robert Kanter
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development