[MESOS-2110] Configurable Ping Timeouts

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.23.0
• Component/s: agent, master
• Labels: master mesosphere network slave timeout
• Target Version/s: 0.23.0
• Sprint: Mesosphere Q4 Sprint 3 - 12/7, Mesosphere Q1 Sprint 1 - 1/23, Mesosphere Q1 Sprint 2 - 2/6, Mesosphere Q1 Sprint 3 - 2/20, Mesosphere Q1 Sprint 4 - 3/6, Mesosphere Q1 Sprint 5 - 3/20, Mesosphere Q1 Sprint 6 - 4/3, Mesosphere Q1 Sprint 7 - 4/17, Mesosphere Q2 Sprint 8 - 5/1, Mesosphere Q1 Sprint 9 - 5/15, Mesosphere Sprint 10, Mesosphere Sprint 11, Mesosphere Sprint 13
• Story Points: 8
• Assignee: Adam B
• Reporter: Adam B
• Shepherd: Niklas Quarfot Nielsen
• Created: 14/Nov/14 02:11
• Updated: 06/Jul/15 18:55
• Resolved: 06/Jul/15 18:55

Description

After a series of ping-failures, the master considers the slave lost and calls shutdownSlave, requiring such a slave that reconnects to kill its tasks and re-register as a new slaveId. On the other side, after a similar timeout, the slave will consider the master lost and try to detect a new master. These timeouts are currently hardcoded constants (5 * 15s), which may not be well-suited for all scenarios.

• Some clusters may tolerate a longer slave process restart period, and wouldn't want tasks to be killed upon reconnect.
• Some clusters may have higher-latency networks (e.g. cross-datacenter, or for volunteer computing efforts), and would like to tolerate longer periods without communication.

We should provide flags/mechanisms on the master to control its tolerance for non-communicative slaves, and (less importantly?) on the slave to tolerate missing masters.

Issue Links

• duplicates: MESOS-237 Make ping timeout configurable on the command line and update the default
• relates to: MESOS-1529 Handle a network partition between Master and Slave