[SPARK-11315] Add YARN extension service to publish Spark events to YARN timeline service

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.5.1
• Component/s: YARN

Environment

Hadoop 2.6+

• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 26/Oct/15 13:44
• Updated: 12/Feb/19 20:45
• Resolved: 12/Feb/19 20:45

Description

Add an extension service (using SPARK-11314) to subscribe to Spark lifecycle events, batch them and forward them to the YARN Application Timeline Service. This data can then be retrieved by a new back end for the Spark History Service, and by other analytics tools.

Issue Links

• links to: [Github] Pull Request #8744 (steveloughran)