Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Incomplete
-
2.2.1, 2.3.0
-
None
-
Description
Two changes in this PR:
- A /health endpoint for a quick binary indication on the health of MesosClusterDispatcher. Useful for those running MesosClusterDispatcher as a marathon app: http://mesosphere.github.io/marathon/docs/health-checks.html. Returns a 503 status if the server is unhealthy and a 200 if the server is healthy
- A /status endpoint for a more detailed examination on the current state of a MesosClusterDispatcher instance. Useful as a troubleshooting/monitoring tool
For both endpoints, regardless of status code, the following body is returned:
{
"action" : "ServerStatusResponse",
"launchedDrivers" : 0,
"message" : "iamok",
"queuedDrivers" : 0,
"schedulerDriverStopped" : false,
"serverSparkVersion" : "2.3.1-SNAPSHOT",
"success" : true,
"pendingRetryDrivers" : 0
}
Aside from surfacing all of the scheduler metrics, the response also includes the status of the Mesos SchedulerDriver. On numerous occasions now, we have observed scenarios where the Mesos SchedulerDriver quietly exits due to some other failure. When this happens, jobs queue up and the only way to clean things up is to restart the service.
With the above health check, marathon can be configured to automatically restart the MesosClusterDispatcher service when the health check fails, lessening the need for manual intervention.
