[SPARK-24501] Add metrics for Mesos Driver and Dispatcher

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.3.1
• Component/s: Mesos
• Labels: bulk-closed
• Flags: Patch
• Assignee: Unassigned
• Reporter: Nicholas Parker
• Created: 08/Jun/18 23:41
• Updated: 25/May/21 01:50
• Resolved: 25/May/21 01:39

Description

The Mesos Dispatcher currently only has three gauge metrics, for the number of waiting, launched, and retrying drivers, defined in MesosClusterSchedulerSource.scala. Meanwhile, the Mesos Driver doesn't have any metrics defined at all.

Implement additional metrics for both the Mesos Dispatcher and Mesos Driver, including e.g. types of events received, how long it takes jobs to run, for use by operators to monitor and/or diagnose their Spark environment.

Issue Links

• links to: [Github] Pull Request #21516 (nickbp)
• links to: GitHub Pull Request #21516