[MESOS-540] Executor health checking.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: health-check twitter
• Assignee: Unassigned
• Reporter: Benjamin Mahler
• Created: 08/Jul/13 21:51
• Updated: 29/Dec/17 03:45

Description

We currently do not health check running executors.

At Twitter, this has led to out-of-band health checking of executors for an internal framework.

For the Storm framework, this has led to out-of-band health checking via ZooKeeper. Health checking would allow Storm to use finer grained executors for better isolation.

This also helps the Hadoop and Jenkins frameworks as well should health checking be desired.

As for implementation, I would propose adding a call on the Executor interface:

/**

• Invoked by the ExecutorDriver to determine the health of the executor.
• When this function returns, the Executor is considered healthy.
  */
  void heartbeat(ExecutorDriver* driver) = 0;

The driver can then heartbeat periodically and kill when the Executor is not responding to heartbeats. The driver should also detect the executor deadlocking on any of the other callbacks.

Issue Links

• is related to: MESOS-7564 Introduce a heartbeat mechanism for v1 HTTP executor <-> agent communication.