[MESOS-10084] Detecting whether executor is generated for command task should work when the launcher_dir changes

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.4, 1.6.3, 1.7.3, 1.8.2, 1.9.1, 1.10.0
• Target Version/s: 1.5.4, 1.6.3, 1.7.3, 1.8.2, 1.9.1, 1.10.0
• Sprint: Studio 4: RI-22 63, Studio 4: RI-23 64
• Story Points: 3
• Assignee: Benjamin Bannier
• Reporter: Andrei Sekretenko
• Shepherd: Benjamin Mahler
• Created: 31/Dec/19 17:48
• Updated: 27/Jan/20 14:27
• Resolved: 24/Jan/20 11:20

Description

As currently implemented, on recovery Mesos agent determines that the executor is generated for command task by comparing the executor command with a current path to Mesos executor:

https://github.com/apache/mesos/blob/1.7.x/src/slave/slave.cpp#L9635

During upgrade of production cluster we observed this check to break due to the new launcher_dir being different from the one of checkpointed executor.

This can cause problems of various kind: for example, after such upgrade, Mesos master can begin to treat the checkpointed command executors as subject to resource quota.

Design considerations:

• proper solution is to checkpoint the flag indicating whether the executor is a command/docker one.
• for correct upgrade from older Mesos versions, we will need some kind of workaround to detect command executors after upgrade; the workaround logic should be skipped if there is a checkpointed flag.