Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10164

Allow NM to start even when custom resource type not defined

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Do
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: nodemanager
    • Labels:
      None

      Description

      In the custom resource documentation, it tells you to add the number of custom resources to a property called yarn.nodemanager.resource-type.<resource> in a file called node-resources.xml.

      For GPU resources, this would look something like

        <property>
          <name>yarn.nodemanager.resource-type.gpu</name>
          <value>16</value>
        </property>
      

      A corresponding config property must also be in resource-types.xml called yarn.resource-types:

        <property>
          <name>yarn.resource-types</name>
          <value>gpu</value>
          <description>Custom resources to be used for scheduling. </description>
        </property>
      

      If the yarn.nodemanager.resource-type.gpu property exists without the corresponding yarn.resource-types property, the nodemanager fails to start.

      I would like the option to automatically create the node-resources.xml on all new nodes regardless of whether or not the cluster supports GPU resources so that if I deploy a GPU node into an existing cluster that does not (yet) support GPU resources, the nodemanager will at least start. Even though it doesn't support the GPU resource, the other supported resources will still be available to be used by the apps in the cluster.

        Attachments

          Activity

            People

            • Assignee:
              epayne Eric Payne
              Reporter:
              epayne Eric Payne
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: