[SPARK-11373] Add metrics to the History Server and providers

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.6.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 28/Oct/15 15:19
• Updated: 21/May/19 07:15
• Resolved: 21/May/19 07:15

Description

The History server doesn't publish metrics about JVM load or anything from the history provider plugins. This means that performance problems from massive job histories aren't visible to management tools, and nor are any provider-generated metrics such as time to load histories, failed history loads, the number of connectivity failures talking to remote services, etc.

If the history server set up a metrics registry and offered the option to publish its metrics, then management tools could view this data.

1. the metrics registry would need to be passed down to the instantiated ApplicationHistoryProvider, in order for it to register its metrics.
2. if the codahale metrics servlet were registered under a path such as /metrics, the values would be visible as HTML and JSON, without the need for management tools.
3. Integration tests could also retrieve the JSON-formatted data and use it as part of the test suites.

Issue Links

• is related to: SPARK-6951 History server slow startup if the event log directory is large
• relates to: SPARK-19111 S3 Mesos history upload fails silently if too large
• relates to: SPARK-1537 Add integration with Yarn's Application Timeline Server
• links to: [Github] Pull Request #9571 (steveloughran)
• links to: [Github] Pull Request #17747 (steveloughran)