[MESOS-7911] Non-checkpointing framework's tasks should not be marked LOST when agent disconnects.

• Type: Bug
• Status: Accepted
• Priority: Critical
• Resolution: Unresolved
• Labels: foundations reliability
• Epic Link: PARTITION_AWARE-improvements
• Sprint: Mesosphere Sprint 74, Mesosphere Sprint 75
• Story Points: 5
• Assignee: Unassigned
• Reporter: Benjamin Mahler
• Shepherd: Vinod Kone
• Created: 23/Aug/17 19:50
• Updated: 03/Apr/19 17:29

Description

Currently, when framework with checkpointing disabled has tasks running on an agent and that agent disconnects from the master, the master will mark those tasks LOST and remove them from its memory. The assumption is that the agent is disconnecting because it terminated.

However, it's possible that this disconnection occurred due to a transient loss of connectivity and the agent re-connects while never having terminated. This case violates our assumption of there being no unknown tasks to the master:

```
void Master::reconcileKnownSlave(
Slave* slave,
const vector<ExecutorInfo>& executors,
const vector<Task>& tasks)
{
...

// TODO(bmahler): There's an implicit assumption here the slave
// cannot have tasks unknown to the master. This should be the
// case since the causal relationship is:
// slave removes task -> master removes task
// Add error logging for any violations of this assumption!
```

As a result, the tasks would remain on the agent but the master would not know about them!

A more appropriate action here would be:

1. When an agent disconnects, mark the tasks as unreachable.
   1. If the framework is not partition aware, only show it the last known task state.
   2. If the framework is partition aware, let it know that it's now unreachable.
2. If the agent re-connects:
   1. And the agent had restarted, let the non-checkpointing framework know its tasks are GONE/LOST.
   2. If the agent still holds the tasks, the tasks are restored as reachable.
3. If the agent gets removed:
   1. For partition aware non-checkpointing frameworks, let them know the tasks are unreachable.
   2. For non partition aware non-checkpointing frameworks, let them know the tasks are lost and kill them if the agent comes back.