Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4649

mr-jobhistory-daemon.sh needs to be updated post YARN-1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.2-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: jobhistoryserver
    • Labels:
      None

      Description

      Even today, JHS is assuming that YARN_HOME will be same as HADOOP_MAPRED_HOME besides other such assumptions. We need to fix it.

      1. MAPREDUCE-4649-20120911.txt
        8 kB
        Vinod Kumar Vavilapalli
      2. MAPREDUCE-4649-20120912.txt
        8 kB
        Vinod Kumar Vavilapalli
      3. MAPREDUCE-4649-20120923.txt
        12 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          Hemanth Yamijala added a comment -

          On this topic, what is the purpose of mr-jobhistory-daemon ? Is it just for starting the history server ? If yes, then should we really pass in the command to start, like:

          mr-jobhistory-daemon.sh start historyserver

          Can't it just be:

          mr-jobhistory-daemon.sh start

          and 'historyserver' is passed internally ?

          Show
          Hemanth Yamijala added a comment - On this topic, what is the purpose of mr-jobhistory-daemon ? Is it just for starting the history server ? If yes, then should we really pass in the command to start, like: mr-jobhistory-daemon.sh start historyserver Can't it just be: mr-jobhistory-daemon.sh start and 'historyserver' is passed internally ?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          On this topic, what is the purpose of mr-jobhistory-daemon ? Is it just for starting the history server ? If yes, then should we really pass in the command to start, like

          Yes, today it is only for starting the history server. But the script is generic enough like the rest of the hadoop scripts to start arbitrary command

          Can't it just be: mr-jobhistory-daemon.sh start and 'historyserver' is passed internally ?

          Instead I think we should rename it back to a generic mapred-daemon.sh to start arbitrary mapreduce specific daemons. Can you file a ticket?

          Show
          Vinod Kumar Vavilapalli added a comment - On this topic, what is the purpose of mr-jobhistory-daemon ? Is it just for starting the history server ? If yes, then should we really pass in the command to start, like Yes, today it is only for starting the history server. But the script is generic enough like the rest of the hadoop scripts to start arbitrary command Can't it just be: mr-jobhistory-daemon.sh start and 'historyserver' is passed internally ? Instead I think we should rename it back to a generic mapred-daemon.sh to start arbitrary mapreduce specific daemons. Can you file a ticket?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Patch to fix this, was a bit of work.

          Show
          Vinod Kumar Vavilapalli added a comment - Patch to fix this, was a bit of work.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12544725/MAPREDUCE-4649-20120911.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2845//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2845//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544725/MAPREDUCE-4649-20120911.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2845//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2845//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Forgot to add YARN jars for the sake of JHS. Fixing that bug now.

          Show
          Vinod Kumar Vavilapalli added a comment - Forgot to add YARN jars for the sake of JHS. Fixing that bug now.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          This patch depends on HADOOP_YARN_HOME, let's order this after YARN-9.

          Show
          Vinod Kumar Vavilapalli added a comment - This patch depends on HADOOP_YARN_HOME, let's order this after YARN-9 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12544882/MAPREDUCE-4649-20120912.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2847//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2847//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544882/MAPREDUCE-4649-20120912.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2847//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2847//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Updated patch, ready for review.

          Tested together with latest patches at YARN-9 and HADOOP-8794.

          Show
          Vinod Kumar Vavilapalli added a comment - Updated patch, ready for review. Tested together with latest patches at YARN-9 and HADOOP-8794 .
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546225/MAPREDUCE-4649-20120923.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2870//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2870//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546225/MAPREDUCE-4649-20120923.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2870//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2870//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Vinod!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Vinod!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2795 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2795/)
          MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224)

          Result = FAILURE
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2795 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2795/ ) MAPREDUCE-4649 . Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2836 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2836/)
          MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2836 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2836/ ) MAPREDUCE-4649 . Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2773 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2773/)
          MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2773 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2773/ ) MAPREDUCE-4649 . Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1177/)
          MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1177/ ) MAPREDUCE-4649 . Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1208 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1208/)
          MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1208 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1208/ ) MAPREDUCE-4649 . Ensure MapReduce JobHistory Daemon doens't assume HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. Contributed by Vinod K V. (Revision 1390224) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390224 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.sh /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml

            People

            • Assignee:
              Vinod Kumar Vavilapalli
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development