[MESOS-1466] Race between executor exited event and launch task can cause overcommit of resources

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: allocation, master
• Labels: reliability twitter
• Epic Link: Executor Reconciliation
• Sprint: Q3 Sprint 3, Q3 Sprint 4
• Story Points: 8
• Assignee: Meng Zhu
• Reporter: Vinod Kone
• Shepherd: Vinod Kone
• Created: 10/Jun/14 00:32
• Updated: 27/Mar/18 13:30
• Resolved: 27/Mar/18 13:30

Description

The following sequence of events can cause an overcommit

--> Launch task is called for a task whose executor is already running

--> Executor's resources are not accounted for on the master

--> Executor exits and the event is enqueued behind launch tasks on the master

--> Master sends the task to the slave which needs to commit for resources for task and the (new) executor.

--> Master processes the executor exited event and re-offers the executor's resources causing an overcommit of resources.

Issue Links

• blocks: MESOS-1654 Expose ephemeral ports per container as a resource
• is blocked by: MESOS-1718 Command executor can overcommit the agent.
• is related to: MESOS-1720 Slave should send exited executor message when the executor is never launched.
• relates to: MESOS-1674 Kill private_resources and treat 'ephemeral_ports' as a resource.