[SPARK-22190] Add Spark executor task metrics to Dropwizard metrics

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core
• Assignee: Luca Canali
• Reporter: Luca Canali
• Created: 03/Oct/17 13:41
• Updated: 26/Sep/18 17:36
• Resolved: 01/Nov/17 14:40

Description

I would like to propose to expose Spark executor task metrics using the Dropwizard metrics. I have developed a simple implementation and run a few tests using Graphite sink and Grafana visualization and this appears to me a good source of information for monitoring and troubleshooting the progress of Spark jobs. I attach a screenshot of an example graph generated with Grafana.

Issue Links

• is related to: SPARK-25536 executorSource.METRIC read wrong record in Executor.scala Line444
• links to: [Github] Pull Request #19426 (LucaCanali)