[MESOS-9847] Docker executor doesn't wait for status updates to be ack'd before shutting down.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.4, 1.6.3, 1.7.3, 1.8.2, 1.9.1, 1.10.0
• Component/s: executor
• Labels: containerization
• Sprint: Containerization: RI-16 Sp 50, Studio 1: RI-23 64, Studio 1: RI-24 65
• Story Points: 5
• Assignee: Andrei Budnik
• Reporter: Meng Zhu
• Created: 13/Jun/19 23:45
• Updated: 19/Mar/20 22:31
• Resolved: 03/Feb/20 15:46

Description

The docker executor doesn't wait for pending status updates to be acknowledged before shutting down, instead it sleeps for one second and then terminates:

  void _stop()
  {
    // A hack for now ... but we need to wait until the status update
    // is sent to the slave before we shut ourselves down.
    // TODO(tnachen): Remove this hack and also the same hack in the
    // command executor when we have the new HTTP APIs to wait until
    // an ack.
    os::sleep(Seconds(1));
    driver.get()->stop();
  }

This would result in racing between task status update (e.g. TASK_FINISHED) and executor exit. The latter would lead agent generating a `TASK_FAILED` status update by itself, leading to the confusing case where the agent handles two different terminal status updates.

Issue Links

• is related to: MESOS-9887 Race condition between two terminal task status updates for Docker/Command executor.
• relates to: MESOS-8537 Default executor doesn't wait for status updates to be ack'd before shutting down