[YARN-10164] Allow NM to start even when custom resource type not defined

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Do
• Component/s: nodemanager
• Assignee: Eric Payne
• Reporter: Eric Payne
• Created: 25/Feb/20 19:53
• Updated: 25/Jan/21 21:42
• Resolved: 25/Jan/21 21:42

Description

In the custom resource documentation, it tells you to add the number of custom resources to a property called yarn.nodemanager.resource-type.<resource> in a file called node-resources.xml.

For GPU resources, this would look something like

  <property>
    <name>yarn.nodemanager.resource-type.gpu</name>
    <value>16</value>
  </property>

A corresponding config property must also be in resource-types.xml called yarn.resource-types:

  <property>
    <name>yarn.resource-types</name>
    <value>gpu</value>
    <description>Custom resources to be used for scheduling. </description>
  </property>

If the yarn.nodemanager.resource-type.gpu property exists without the corresponding yarn.resource-types property, the nodemanager fails to start.

I would like the option to automatically create the node-resources.xml on all new nodes regardless of whether or not the cluster supports GPU resources so that if I deploy a GPU node into an existing cluster that does not (yet) support GPU resources, the nodemanager will at least start. Even though it doesn't support the GPU resource, the other supported resources will still be available to be used by the apps in the cluster.