Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2437

start-yarn.sh/stop-yarn should give info

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:

      Description

      With the merger and cleanup of the daemon launch code, yarn-daemons.sh no longer prints "Starting" information. This should be made more of an analog of start-dfs.sh/stop-dfs.sh.

      1. YARN-2437.001.patch
        1 kB
        Varun Saxena
      2. YARN-2437.002.patch
        1 kB
        Varun Saxena
      3. YARN-2437.patch
        0.6 kB
        Varun Saxena

        Issue Links

          Activity

          Hide
          haogao Hao Gao added a comment -

          Should the information be printed in start-yarn.sh and stop-yarn.sh?

          Show
          haogao Hao Gao added a comment - Should the information be printed in start-yarn.sh and stop-yarn.sh?
          Hide
          aw Allen Wittenauer added a comment -

          Yeah. Updated the JIRA to reflect that.

          Show
          aw Allen Wittenauer added a comment - Yeah. Updated the JIRA to reflect that.
          Hide
          aw Allen Wittenauer added a comment -

          This code change should be in start and stop and not in yarn. Doing it yarn means that running 'yarn resourcemanager' throws a message which is not ideal for startup scripts.

          Show
          aw Allen Wittenauer added a comment - This code change should be in start and stop and not in yarn. Doing it yarn means that running 'yarn resourcemanager' throws a message which is not ideal for startup scripts.
          Hide
          varun_saxena Varun Saxena added a comment -

          Allen Wittenauer, we cant print hosts information alongside Starting message unlike start-dfs.sh though. Because we do not yet support a command like hdfs getconf -namenodes in YARN

          Show
          varun_saxena Varun Saxena added a comment - Allen Wittenauer , we cant print hosts information alongside Starting message unlike start-dfs.sh though. Because we do not yet support a command like hdfs getconf -namenodes in YARN
          Hide
          aw Allen Wittenauer added a comment -

          Why can't you just use the HDFS one?

          Show
          aw Allen Wittenauer added a comment - Why can't you just use the HDFS one?
          Hide
          varun_saxena Varun Saxena added a comment -

          Allen Wittenauer, you mean why cant we use the same logic which is present in start-dfs.sh to print the list of node managers to be started ?

          Show
          varun_saxena Varun Saxena added a comment - Allen Wittenauer , you mean why cant we use the same logic which is present in start-dfs.sh to print the list of node managers to be started ?
          Hide
          aw Allen Wittenauer added a comment -

          Yup. Go ahead and use 'hdfs getconf' to get the info.

          Show
          aw Allen Wittenauer added a comment - Yup. Go ahead and use 'hdfs getconf' to get the info.
          Hide
          varun_saxena Varun Saxena added a comment -

          Allen Wittenauer, what I meant in my earlier comment was "hdfs getconf" is not supported for YARN. hdfs getconf essentially reads hdfs-site.xml and gets a list of namenodes by using below command :

          hdfs getconf -namenodes
          

          Infact, even "hdfs getconf -datanodes" is not supported.
          So in start-dfs.sh, we can list all the namenodes which are being started but cant do the same for datanodes. So you will find only "Starting datanodes" message when datanodes are started.

          For YARN, as there is no such command, we cant get hosts info. And hence can only print "Starting resourcemanager" or "Starting nodemanagers".
          Even in 2.4, we only printed "starting resourcemanager, logging to xxxxx"
          There is a JIRA open though for implementing "yarn getconf" command.

          Show
          varun_saxena Varun Saxena added a comment - Allen Wittenauer , what I meant in my earlier comment was "hdfs getconf" is not supported for YARN. hdfs getconf essentially reads hdfs-site.xml and gets a list of namenodes by using below command : hdfs getconf -namenodes Infact, even "hdfs getconf -datanodes" is not supported. So in start-dfs.sh, we can list all the namenodes which are being started but cant do the same for datanodes. So you will find only "Starting datanodes" message when datanodes are started. For YARN, as there is no such command, we cant get hosts info. And hence can only print "Starting resourcemanager" or "Starting nodemanagers". Even in 2.4, we only printed "starting resourcemanager, logging to xxxxx" There is a JIRA open though for implementing "yarn getconf" command.
          Hide
          varun_saxena Varun Saxena added a comment -

          We can although read the $

          {CONFIG_DIR}

          /slaves file and get a list of slaves. Currently slaves file is read(contents not printed) in hadoop_connect_to_hosts function in hadoop-functions.sh
          We can read it in yarn-daemons.sh too, if the file exists to provide hosts info.

          Show
          varun_saxena Varun Saxena added a comment - We can although read the $ {CONFIG_DIR} /slaves file and get a list of slaves. Currently slaves file is read(contents not printed) in hadoop_connect_to_hosts function in hadoop-functions.sh We can read it in yarn-daemons.sh too, if the file exists to provide hosts info.
          Hide
          varun_saxena Varun Saxena added a comment -

          I meant read it in start-yarn.sh

          Show
          varun_saxena Varun Saxena added a comment - I meant read it in start-yarn.sh
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12685148/YARN-2437.001.patch
          against trunk revision 7896815.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestHdfsAdmin

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5998//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5998//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685148/YARN-2437.001.patch against trunk revision 7896815. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHdfsAdmin +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5998//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5998//console This message is automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Test failure unrelated to change

          Show
          varun_saxena Varun Saxena added a comment - Test failure unrelated to change
          Hide
          aw Allen Wittenauer added a comment -

          Oh, I get it now.

          One minor nit: proxyservers is commented out so we shouldn't print a message for it.

          Show
          aw Allen Wittenauer added a comment - Oh, I get it now. One minor nit: proxyservers is commented out so we shouldn't print a message for it.
          Hide
          varun_saxena Varun Saxena added a comment -

          Allen Wittenauer, kindly check the latest patch. I guess it should be fine now.

          Show
          varun_saxena Varun Saxena added a comment - Allen Wittenauer , kindly check the latest patch. I guess it should be fine now.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12685416/YARN-2437.002.patch
          against trunk revision 3c72f54.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6007//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6007//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685416/YARN-2437.002.patch against trunk revision 3c72f54. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6007//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6007//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Looks good to me.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Looks good to me.
          Hide
          aw Allen Wittenauer added a comment -

          Agree.

          +1 will commit to trunk.

          Thanks!

          Show
          aw Allen Wittenauer added a comment - Agree. +1 will commit to trunk. Thanks!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6691/)
          YARN-2437. start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6691/ ) YARN-2437 . start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #35 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/35/)
          YARN-2437. start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #35 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/35/ ) YARN-2437 . start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1969 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1969/)
          YARN-2437. start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7)

          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1969 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1969/ ) YARN-2437 . start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7) hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #39 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/39/)
          YARN-2437. start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #39 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/39/ ) YARN-2437 . start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1989 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1989/)
          YARN-2437. start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7)

          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1989 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1989/ ) YARN-2437 . start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) (aw: rev 59cb8b9123fac725660fc7cfbaaad3d1aa3e3bd7) hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              varun_saxena Varun Saxena
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development