Hadoop Common
  1. Hadoop Common
  2. HADOOP-8393

hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If you start a pseudo distributed yarn using "start-yarn.sh" you need to specify exports for HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, YARN_HOME, YARN_CONF_DIR, and HADOOP_MAPRED_HOME in hadoop-env.sh (or elsewhere), otherwise the spawned node manager will be missing these in it's environment. This is due to start-yarn using yarn-daemons. With this fix it's possible to start yarn (etc...) with only HADOOP_CONF_DIR specified in the environment. Took some time to track down this failure, so seems worthwhile to fix.

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322473 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #304 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/304/)
          svn merge -c 1338998 FIXES: HADOOP-8393 hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1356763)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356763
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #304 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/304/ ) svn merge -c 1338998 FIXES: HADOOP-8393 hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1356763) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356763 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Robert Joseph Evans made changes -
          Fix Version/s 0.23.3 [ 12320059 ]
          Fix Version/s 2.0.1-alpha [ 12321441 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1081 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1081/)
          HADOOP-8393. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1081 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1081/ ) HADOOP-8393 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1045 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1045/)
          HADOOP-8393. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1045 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1045/ ) HADOOP-8393 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2267 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2267/)
          HADOOP-8393. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998)

          Result = ABORTED
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2267 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2267/ ) HADOOP-8393 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2250/)
          HADOOP-8393. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2250/ ) HADOOP-8393 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2324 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2324/)
          HADOOP-8393. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2324 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2324/ ) HADOOP-8393 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1338998) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338998 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.1 [ 12321441 ]
          Resolution Fixed [ 1 ]
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Patrick. committed to trunk and branch-2.

          Show
          Alejandro Abdelnur added a comment - Thanks Patrick. committed to trunk and branch-2.
          Hide
          Alejandro Abdelnur added a comment -

          +1 lgtm

          Show
          Alejandro Abdelnur added a comment - +1 lgtm
          Hide
          Ahmed Radwan added a comment -

          lgtm +1, thanks Patrick!

          Show
          Ahmed Radwan added a comment - lgtm +1, thanks Patrick!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12526559/HADOOP-8393.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/985//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/985//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526559/HADOOP-8393.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/985//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/985//console This message is automatically generated.
          Patrick Hunt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Patrick Hunt made changes -
          Attachment HADOOP-8393.patch [ 12526559 ]
          Hide
          Patrick Hunt added a comment -

          Exported the necessary vars. I tested by packaging everything up, starting start-dfs and start-yarn (with a single "localhost" slave) and then running a pi job from examples. Works with the patch, doesn't w/o.

          Show
          Patrick Hunt added a comment - Exported the necessary vars. I tested by packaging everything up, starting start-dfs and start-yarn (with a single "localhost" slave) and then running a pi job from examples. Works with the patch, doesn't w/o.
          Patrick Hunt made changes -
          Field Original Value New Value
          Link This issue incorporates MAPREDUCE-4250 [ MAPREDUCE-4250 ]
          Patrick Hunt created issue -

            People

            • Assignee:
              Patrick Hunt
              Reporter:
              Patrick Hunt
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development