[YARN-9217] Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.0.0, 3.1.0
Fix Version/s: 3.3.0, 3.2.1
Component/s: yarn
Labels:
None

Hadoop Flags:

Reviewed

Description

Nodemanager will not start

1. If Autodiscovery is enabled:

If nvidia-smi path is misconfigured or the file does not exist.
There is 0 GPU found
If the file exists but it is not pointing to an nvidia-smi
if the binary is ok but there is an IOException

2. If the manually configured GPU devices are misconfigured

Any index:minor number format failure will cause a problem
0 configured device will cause a problem
NumberFormatException is not handled

It would be a better option to add warnings about the configuration, set 0 available GPUs and let the node work and run non-gpu jobs.

Attachments

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

YARN-9217.branch-3.2.001.patch
16/Aug/19 14:09
40 kB
Peter Bacsko
YARN-9217.012.patch
16/Aug/19 11:59
39 kB
Peter Bacsko
YARN-9217.011.patch
15/Aug/19 12:54
40 kB
Peter Bacsko
YARN-9217.010.patch
15/Aug/19 10:37
39 kB
Peter Bacsko
YARN-9217.009.patch
13/Aug/19 15:15
39 kB
Peter Bacsko
YARN-9217.008.patch
09/Aug/19 10:05
39 kB
Peter Bacsko
YARN-9217.007.patch
09/Aug/19 08:29
39 kB
Peter Bacsko
YARN-9217.006.patch
07/Aug/19 11:15
39 kB
Peter Bacsko
YARN-9217.005.patch
20/Jul/19 13:49
39 kB
Peter Bacsko
YARN-9217.004.patch
07/Feb/19 13:43
43 kB
Antal Bálint Steinbach
YARN-9217.003.patch
06/Feb/19 15:49
39 kB
Antal Bálint Steinbach
YARN-9217.002.patch
06/Feb/19 14:53
36 kB
Antal Bálint Steinbach
YARN-9217.001.patch
06/Feb/19 10:31
30 kB
Antal Bálint Steinbach

Activity

People

Assignee:: Peter Bacsko

Reporter:: Antal Bálint Steinbach

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 22/Jan/19 10:22

Updated:: 21/Aug/19 16:33

Resolved:: 21/Aug/19 14:51