[MESOS-8038] Launching GPU task sporadically fails. - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Reviewable
Priority: Critical
Resolution: Unresolved
Affects Version/s: 1.4.0
Fix Version/s: None
Component/s: containerization, gpu
Labels:
None

Description

I was running a job which uses GPUs. It runs fine most of the time.
But occasionally I see the following message in the mesos log.
"Collect failed: Requested 1 but only 0 available"
Followed by executor getting killed and the tasks getting lost. This happens even before the the job starts. A little search in the code base points me to something related to GPU resource being the probable cause.

There is no deterministic way that this can be reproduced. It happens occasionally.
I have attached the slave log for the issue.

Using 1.4.0 Mesos Master and 1.4.0 Mesos Slave.