Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10123

Error message around yarn app -stop/start can be improved to highlight that an implementation at framework level is needed for the stop/start functionality to work

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.2.1
    • 3.4.0, 3.2.3, 3.3.2
    • client, documentation
    • None

    Description

      A "stop" on a YARN application fails with the below error:

      # yarn app -stop application_1581294743321_0002 -appTypes SPARK
      20/02/10 06:24:27 INFO client.RMProxy: Connecting to ResourceManager at c3224-node2.squadron.support.hortonworks.com/172.25.34.128:8050
      20/02/10 06:24:27 INFO client.AHSProxy: Connecting to Application History server at c3224-node2.squadron.support.hortonworks.com/172.25.34.128:10200
      Exception in thread "main" java.lang.IllegalArgumentException: App admin client class name not specified for type SPARK
              at org.apache.hadoop.yarn.client.api.AppAdminClient.createAppAdminClient(AppAdminClient.java:76)
              at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:579)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
              at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:123)
      

      From https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java#L76, it seems that this is because user does not have the setting:

      yarn.application.admin.client.class.SPARK
      

      set up in their client configuration.

      However, even if this setting is present, we still need to have an implementation available for the application type. From my internal discussions - Jobs don't have a notion of stop / resume functionality at YARN level. If some apps like Spark need it, it has to be implemented at those framework's level.

      Therefore, the above error message is a bit misleading in that, even if "yarn.application.admin.client.class.SPARK" is supplied (or for that matter - yarn.application.admin.client.class.MAPREDUCE), if there is no implementation actually available underneath to handle the stop/start functionality then, we will fail again, albeit with a different error here: https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java#L85.

      As such, maybe this error message can be potentially improved to say something like:

      Exception in thread "main" java.lang.IllegalArgumentException: App admin client class name not specified for type SPARK. Please ensure the App admin client class actually exists within SPARK to handle this functionality.
      

      or something similar.

      Further, documentation around "-stop" and "-start" options will need to be improved here -> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application_or_app as it does not mention anything about having an implementation at the framework level for the YARN stop/start command to succeed.

      Attachments

        1. YARN-10123.branch-3.3.001.patch
          9 kB
          Siddharth Ahuja
        2. YARN-10123.branch-3.2.001.patch
          9 kB
          Siddharth Ahuja
        3. YARN-10123.001.patch
          9 kB
          Siddharth Ahuja

        Activity

          People

            sahuja Siddharth Ahuja
            sahuja Siddharth Ahuja
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: