Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3869

Distributed shell application fails with NoClassDefFoundError

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: 2.0.0-alpha
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Distributed shell application always fails to start the application master with the following error.

      12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running ApplicationMaster
      java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
      	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.<init>(ApplicationMaster.java:252)
      	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.ipc.YarnRPC
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
      	... 2 more
      
      1. MAPREDUCE-3869.patch
        1 kB
        Devaraj K
      2. MAPREDUCE-3869-1.patch
        2 kB
        Devaraj K
      3. MAPREDUCE-3869-2.patch
        2 kB
        Devaraj K

        Issue Links

          Activity

          Hide
          Devaraj K added a comment -

          Here client is setting the classpath environment with $YARN_HOME/modules/* and $YARN_HOME/lib/* but those are not present and causing the NoClassDefFoundError.

          I have provided trivial and direct patch which fixes this issue.

          Show
          Devaraj K added a comment - Here client is setting the classpath environment with $YARN_HOME/modules/* and $YARN_HOME/lib/* but those are not present and causing the NoClassDefFoundError. I have provided trivial and direct patch which fixes this issue.
          Hide
          Devaraj K added a comment -
          org.apache.hadoop.yarn.applications.distributedshell.Client
                  + ":$HADOOP_HDFS_HOME/share/hadoop/hdfs/*"
                  + ":$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*"
                  + ":$YARN_HOME/modules/*"
                  + ":$YARN_HOME/lib/*"
                  + ":./log4j.properties:";
          
          Show
          Devaraj K added a comment - org.apache.hadoop.yarn.applications.distributedshell.Client + ":$HADOOP_HDFS_HOME/share/hadoop/hdfs/*" + ":$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*" + ":$YARN_HOME/modules/*" + ":$YARN_HOME/lib/*" + ":./log4j.properties:" ;
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12514783/MAPREDUCE-3869.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1880//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1880//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514783/MAPREDUCE-3869.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1880//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1880//console This message is automatically generated.
          Hide
          Hitesh Shah added a comment -

          @Devaraj, would it be better to change the code to use YarnConfiguration.YARN_APPLICATION_CLASSPATH instead?

          Show
          Hitesh Shah added a comment - @Devaraj, would it be better to change the code to use YarnConfiguration.YARN_APPLICATION_CLASSPATH instead?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          would it be better to change the code to use YarnConfiguration.YARN_APPLICATION_CLASSPATH instead?

          +1. Devaraj, can you please do that? Tx.

          Show
          Vinod Kumar Vavilapalli added a comment - would it be better to change the code to use YarnConfiguration.YARN_APPLICATION_CLASSPATH instead? +1. Devaraj, can you please do that? Tx.
          Hide
          Devaraj K added a comment -

          Thanks Hitesh and Vinod for looking into the patch.

          I'm sorry for the late response.

          I have updated the patch with the suggested change.

          Show
          Devaraj K added a comment - Thanks Hitesh and Vinod for looking into the patch. I'm sorry for the late response. I have updated the patch with the suggested change.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12518022/MAPREDUCE-3869-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2037//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2037//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-applications-distributedshell.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2037//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518022/MAPREDUCE-3869-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2037//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2037//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-applications-distributedshell.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2037//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12518171/MAPREDUCE-3869-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2043//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2043//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518171/MAPREDUCE-3869-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2043//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2043//console This message is automatically generated.
          Hide
          Devaraj K added a comment -

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          It is classpath issue. I have verified it manually. No tests needed for this.

          Show
          Devaraj K added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. It is classpath issue. I have verified it manually. No tests needed for this.
          Hide
          Siddharth Seth added a comment -

          +1. Will commit this shortly.

          Show
          Siddharth Seth added a comment - +1. Will commit this shortly.
          Hide
          Siddharth Seth added a comment -

          Committed to trunk and branch-2. Thanks Devaraj.

          Show
          Siddharth Seth added a comment - Committed to trunk and branch-2. Thanks Devaraj.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2032 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2032/)
          MAPREDUCE-3869. Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523)

          Result = SUCCESS
          sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2032 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2032/ ) MAPREDUCE-3869 . Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2107 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2107/)
          MAPREDUCE-3869. Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523)

          Result = SUCCESS
          sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2107 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2107/ ) MAPREDUCE-3869 . Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2044 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2044/)
          MAPREDUCE-3869. Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523)

          Result = ABORTED
          sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2044 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2044/ ) MAPREDUCE-3869 . Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523) Result = ABORTED sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1010 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1010/)
          MAPREDUCE-3869. Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523)

          Result = FAILURE
          sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1010 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1010/ ) MAPREDUCE-3869 . Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1045 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1045/)
          MAPREDUCE-3869. Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523)

          Result = SUCCESS
          sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1045 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1045/ ) MAPREDUCE-3869 . Fix classpath for DistributedShell application. (Contributed by Devaraj K) (Revision 1311523) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311523 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          Hide
          Kaibo Zhou added a comment -

          This problem appear when i install 0.23.1 and run the example.
          I found the hadoop-yarn-common-0.23.1.jar which contain org/apache/hadoop/yarn/ipc/YarnRPC is under $HADOOP_HDFS_HOME/share/hadoop/mapreduce/ dir.
          or shoud add $HADOOP_HDFS_HOME/share/hadoop/mapreduce/ in Client.java?

          Show
          Kaibo Zhou added a comment - This problem appear when i install 0.23.1 and run the example. I found the hadoop-yarn-common-0.23.1.jar which contain org/apache/hadoop/yarn/ipc/YarnRPC is under $HADOOP_HDFS_HOME/share/hadoop/mapreduce/ dir. or shoud add $HADOOP_HDFS_HOME/share/hadoop/mapreduce/ in Client.java?

            People

            • Assignee:
              Devaraj K
              Reporter:
              Devaraj K
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development