Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4097

tools testcases fail because missing mrapp-generated-classpath file in classpath

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The mrapp-generated-classpath file is created in hadoop-mapreduce-client-apptarget/classes/ dir but it is excluded from the JAR.

      When running tools testcases from root level, mvn uses hadoop-mapreduce-client-app/target/classes/ dir to create the classpath.

      When running tools testcases from tools level, mvn uses the hadoop-mapreduce-client-app JAR from M2 cache to create the classpath.

      In the later the mrapp-generated-classpath is not present.

      1. MAPREDUCE-4097.trunk-patch.txt
        2 kB
        Roman Shaposhnik
      2. MAPREDUCE-4097.patch.txt
        2 kB
        Roman Shaposhnik
      3. MAPREDUCE-4097.patch
        2 kB
        Alejandro Abdelnur

        Activity

        Alejandro Abdelnur created issue -
        Hide
        Alejandro Abdelnur added a comment -

        The patch creates locally in each subproject the mrapp-generated-classpath file (and exclude it from the JAR). moving the logic to create the mrapp-generated-classpath file and excluded it from the JAR to hadoop-project/pom.xml so everybody gets it.

        did a 'mvn install -DskipTests' at root level.

        did a 'mvn test' at hadoop-tools level, all tests run successfully.

        Show
        Alejandro Abdelnur added a comment - The patch creates locally in each subproject the mrapp-generated-classpath file (and exclude it from the JAR). moving the logic to create the mrapp-generated-classpath file and excluded it from the JAR to hadoop-project/pom.xml so everybody gets it. did a 'mvn install -DskipTests' at root level. did a 'mvn test' at hadoop-tools level, all tests run successfully.
        Alejandro Abdelnur made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-4097.patch [ 12521049 ]
        Alejandro Abdelnur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521049/MAPREDUCE-4097.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2130//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521049/MAPREDUCE-4097.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2130//console This message is automatically generated.
        Hide
        Harsh J added a comment -

        +1, looks good to me. I applied and ran 'mvn test' at hadoop-tools level.

        Show
        Harsh J added a comment - +1, looks good to me. I applied and ran 'mvn test' at hadoop-tools level.
        Hide
        Alejandro Abdelnur added a comment -

        patch seems to have an issue, when using the -Psrc profile the common JAR files as MIA from the hadoop-dist.

        Roman is investigating this, assigning JIRA to him.

        Show
        Alejandro Abdelnur added a comment - patch seems to have an issue, when using the -Psrc profile the common JAR files as MIA from the hadoop-dist. Roman is investigating this, assigning JIRA to him.
        Alejandro Abdelnur made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Assignee Alejandro Abdelnur [ tucu00 ] Roman Shaposhnik [ rvs ]
        Hide
        Roman Shaposhnik added a comment -

        Lets try this one!

        Show
        Roman Shaposhnik added a comment - Lets try this one!
        Roman Shaposhnik made changes -
        Attachment MAPREDUCE-4097.patch.txt [ 12521276 ]
        Roman Shaposhnik made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521276/MAPREDUCE-4097.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2142//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521276/MAPREDUCE-4097.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2142//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        Roman, the patch does not apply correctly

        Show
        Alejandro Abdelnur added a comment - Roman, the patch does not apply correctly
        Hide
        Roman Shaposhnik added a comment -

        @Alejandro, it does to branch-0.23

        Which branch do you want me to cut a patch for?

        Show
        Roman Shaposhnik added a comment - @Alejandro, it does to branch-0.23 Which branch do you want me to cut a patch for?
        Hide
        Roman Shaposhnik added a comment -

        Attaching a patch that applies cleanly on trunk. Let me know if patches for any other branches are desirable.

        Show
        Roman Shaposhnik added a comment - Attaching a patch that applies cleanly on trunk. Let me know if patches for any other branches are desirable.
        Roman Shaposhnik made changes -
        Attachment MAPREDUCE-4097.trunk-patch.txt [ 12521345 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521345/MAPREDUCE-4097.trunk-patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2149//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521345/MAPREDUCE-4097.trunk-patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2149//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        Thanks Roman. Committed to trunk and branch-2

        Show
        Alejandro Abdelnur added a comment - Thanks Roman. Committed to trunk and branch-2
        Alejandro Abdelnur made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2073 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2073/)
        MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2073 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2073/ ) MAPREDUCE-4097 . tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1998 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1998/)
        MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1998 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1998/ ) MAPREDUCE-4097 . tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2010 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2010/)
        MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2010 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2010/ ) MAPREDUCE-4097 . tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/)
        MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/ ) MAPREDUCE-4097 . tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/)
        MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/ ) MAPREDUCE-4097 . tools testcases fail because missing mrapp-generated-classpath file in classpath (rvs via tucu) (Revision 1309583) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309583 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Robert Joseph Evans added a comment -

        I pulled this into 0.23.3

        Show
        Robert Joseph Evans added a comment - I pulled this into 0.23.3
        Robert Joseph Evans made changes -
        Fix Version/s 0.23.3 [ 12320060 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #350 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/350/)
        svn merge -c 1307124, 1307623, 1309583 FIXES: MAPREDUCE-4082 MAPREDUCE-4091 and MAPREDUCE-4097 hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR (tucu) (Revision 1375190)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375190
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #350 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/350/ ) svn merge -c 1307124, 1307623, 1309583 FIXES: MAPREDUCE-4082 MAPREDUCE-4091 and MAPREDUCE-4097 hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR (tucu) (Revision 1375190) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375190 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322471 ]
        Fix Version/s 2.0.0-alpha [ 12320354 ]

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Alejandro Abdelnur
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development