[YARN-9217] Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.0
• Fix Version/s: 3.3.0, 3.2.1
• Component/s: yarn
• Hadoop Flags: Reviewed
• Assignee: Peter Bacsko
• Reporter: Antal Bálint Steinbach
• Created: 22/Jan/19 10:22
• Updated: 21/Aug/19 16:33
• Resolved: 21/Aug/19 14:51

Description

Nodemanager will not start

1. If Autodiscovery is enabled:

• If nvidia-smi path is misconfigured or the file does not exist.
• There is 0 GPU found
• If the file exists but it is not pointing to an nvidia-smi
• if the binary is ok but there is an IOException

2. If the manually configured GPU devices are misconfigured

• Any index:minor number format failure will cause a problem
• 0 configured device will cause a problem
• NumberFormatException is not handled

It would be a better option to add warnings about the configuration, set 0 available GPUs and let the node work and run non-gpu jobs.