Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23943

Improve observability of MesosRestServer/MesosClusterDispatcher

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.2.1, 2.3.0
    • Fix Version/s: None
    • Component/s: Deploy, Mesos
    • Labels:
      None
    • Environment:

       

       

      Description

      Two changes in this PR:

      • A /health endpoint for a quick binary indication on the health of MesosClusterDispatcher. Useful for those running MesosClusterDispatcher as a marathon app: http://mesosphere.github.io/marathon/docs/health-checks.html. Returns a 503 status if the server is unhealthy and a 200 if the server is healthy
      • A /status endpoint for a more detailed examination on the current state of a MesosClusterDispatcher instance. Useful as a troubleshooting/monitoring tool

      For both endpoints, regardless of status code, the following body is returned:

       

      {
        "action" : "ServerStatusResponse",
        "launchedDrivers" : 0,
        "message" : "iamok",
        "queuedDrivers" : 0,
        "schedulerDriverStopped" : false,
        "serverSparkVersion" : "2.3.1-SNAPSHOT",
        "success" : true,
        "pendingRetryDrivers" : 0
      }

      Aside from surfacing all of the scheduler metrics, the response also includes the status of the Mesos SchedulerDriver. On numerous occasions now, we have observed scenarios where the Mesos SchedulerDriver quietly exits due to some other failure. When this happens, jobs queue up and the only way to clean things up is to restart the service. 

      With the above health check, marathon can be configured to automatically restart the MesosClusterDispatcher service when the health check fails, lessening the need for manual intervention.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              adobe_pmackles paul mackles
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: