[SPARK-24550] Add support for Kubernetes specific metrics

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.1
• Component/s: Kubernetes, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Stavros Kontopoulos
• Created: 13/Jun/18 13:18
• Updated: 17/May/20 18:26
• Resolved: 08/Oct/19 05:42

Description

Spark by default offers a specific set of metrics for monitoring. It is possible to add platform specific metrics or enhance the existing ones if it makes sense. Here is an example for mesos. Some example of metrics that could be added are: number of blacklisted nodes, utilization of executors per node, evicted pods and driver restarts, back-off time when requesting executors etc...