[MESOS-9667] Check failure when executor for task using resource provider resources subscribes before agent is registered

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.8.0, 1.9.0
• Component/s: agent
• Labels: foundations mesosphere mesosphere-dss-ga
• Target Version/s: 1.8.0
• Sprint: Storage: RI-12 Sprint 43
• Story Points: 3
• Assignee: Benjamin Bannier
• Reporter: Benjamin Bannier
• Shepherd: Chun-Hung Hsiao
• Created: 21/Mar/19 17:03
• Updated: 03/May/19 03:46
• Resolved: 09/Apr/19 20:04

Description

When an executor for a task using resource provider resources subscribes before the agent has registered with the master, we trigger a fatal assertion,

Mar 21 13:42:47 agent1 mesos-agent[17277]: F0321 13:42:46.845535 17295 slave.cpp:8834] Check failed: 'resourceProviderManager.get()' Must be non NULL
Mar 21 13:42:47 agent1 mesos-agent[17277]: *** Check failure stack trace: *

The reason for this failure is that we attempt to publish resources to the resource provider via the resource provider manager, but the resource provider manager is only created once the agent has registered with the master.

As a workaround one can terminate the executors and their tasks, and let the framework relaunch the tasks (provided it supports that).

A possible workaround could be to prevent such executors from subscribing until the resource provider manager is available.

Issue Links

• is related to: MESOS-6274 Agent should not allow HTTP executors to re-subscribe before containerizer recovery is done.
• relates to: MESOS-9711 Avoid shutting down executors registering before a required resource provider.