[MESOS-8038] Launching GPU task sporadically fails.

• Type: Bug
• Status: Reviewable
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Component/s: containerization, gpu
• Assignee: Zhitao Li
• Reporter: Sai Teja Ranuva
• Shepherd: Gilbert Song
• Created: 28/Sep/17 19:03
• Updated: 10/Jun/21 07:14

Description

I was running a job which uses GPUs. It runs fine most of the time.
But occasionally I see the following message in the mesos log.
"Collect failed: Requested 1 but only 0 available"
Followed by executor getting killed and the tasks getting lost. This happens even before the the job starts. A little search in the code base points me to something related to GPU resource being the probable cause.

There is no deterministic way that this can be reproduced. It happens occasionally.
I have attached the slave log for the issue.

Using 1.4.0 Mesos Master and 1.4.0 Mesos Slave.

Issue Links

• is duplicated by: MESOS-10119 failure to destroy container can cause the agent to "leak" a GPU