Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4250

hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: nodemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is the MR side of HADOOP-8393

      If you start a pseudo distributed yarn using "start-yarn.sh" you need to specify exports for HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, YARN_HOME, YARN_CONF_DIR, and HADOOP_MAPRED_HOME in hadoop-env.sh (or elsewhere), otherwise the spawned node manager will be missing
      these in it's environment. This is due to start-yarn using yarn-daemons. With this fix it's possible to start yarn (etc...) with only HADOOP_CONF_DIR specified in the environment. Took some time to track down this failure, so seems worthwhile to fix.

        Issue Links

          Activity

          Hide
          Patrick Hunt added a comment -

          Exported the necessary vars. I tested by packaging everything up, starting start-dfs and start-yarn (with a single "localhost" slave) and then running a pi job from examples. Works with the patch, doesn't w/o.

          Show
          Patrick Hunt added a comment - Exported the necessary vars. I tested by packaging everything up, starting start-dfs and start-yarn (with a single "localhost" slave) and then running a pi job from examples. Works with the patch, doesn't w/o.
          Hide
          Trevor Robinson added a comment -

          +1 on the issue and the fix. This was tripping me up too.

          Show
          Trevor Robinson added a comment - +1 on the issue and the fix. This was tripping me up too.
          Hide
          Ahmed Radwan added a comment -

          +1 for the patch. Thanks Patrick!

          Show
          Ahmed Radwan added a comment - +1 for the patch. Thanks Patrick!
          Hide
          Alejandro Abdelnur added a comment -

          +1 lgtm

          Show
          Alejandro Abdelnur added a comment - +1 lgtm
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Patrick. committed to trunk and branch-2.

          Show
          Alejandro Abdelnur added a comment - Thanks Patrick. committed to trunk and branch-2.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2324 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2324/)
          MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2324 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2324/ ) MAPREDUCE-4250 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2250/)
          MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2250 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2250/ ) MAPREDUCE-4250 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2267 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2267/)
          MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000)

          Result = ABORTED
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2267 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2267/ ) MAPREDUCE-4250 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1045 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1045/)
          MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1045 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1045/ ) MAPREDUCE-4250 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1081 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1081/)
          MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1081 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1081/ ) MAPREDUCE-4250 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #304 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/304/)
          svn merge -c 1339000. FIXES: MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1356764)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356764
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #304 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/304/ ) svn merge -c 1339000. FIXES: MAPREDUCE-4250 . hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1356764) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356764 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh

            People

            • Assignee:
              Patrick Hunt
              Reporter:
              Patrick Hunt
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development