[SPARK-1537] Add integration with Yarn's Application Timeline Server

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: YARN
• Assignee: Unassigned
• Reporter: Marcelo Masiero Vanzin
• Created: 19/Apr/14 00:02
• Updated: 22/Apr/20 01:03
• Resolved: 12/Feb/19 21:42

Description

It would be nice to have Spark integrate with Yarn's Application Timeline Server (see YARN-321, YARN-1530). This would allow users running Spark on Yarn to have a single place to go for all their history needs, and avoid having to manage a separate service (Spark's built-in server).

At the moment, there's a working version of the ATS in the Hadoop 2.4 branch, although there is still some ongoing work. But the basics are there, and I wouldn't expect them to change (much) at this point.

Issue Links

• depends upon: HADOOP-11822 mark org.apache.hadoop.security.ssl.SSLFactory as @Public
• depends upon: YARN-3539 Compatibility doc to state that ATS v1 is a stable REST API
• is related to: SPARK-11373 Add metrics to the History Server and providers
• relates to: YARN-1530 [Umbrella] Store, manage and serve per-framework application-timeline data
• links to: [Github] Pull Request #4683 (zhzhan)
• links to: [Github] Pull Request #5423 (steveloughran)
• links to: [Github] Pull Request #8744 (steveloughran)
• links to: [Github] Pull Request #9182 (steveloughran)
• links to: [Github] Pull Request #10545 (steveloughran)

Sub-Tasks

• Add service API and test service for Yarn Cluster schedulers: Resolved
• Add YARN extension service to publish Spark events to YARN timeline service: Resolved
• Add History Service Provider to service application histories from YARN timeline server: Resolved