Type: New Feature
Status: In Progress
Affects Version/s: None
Fix Version/s: None
It would be nice to have Spark integrate with Yarn's Application Timeline Server (see
YARN-321, YARN-1530). This would allow users running Spark on Yarn to have a single place to go for all their history needs, and avoid having to manage a separate service (Spark's built-in server).
At the moment, there's a working version of the ATS in the Hadoop 2.4 branch, although there is still some ongoing work. But the basics are there, and I wouldn't expect them to change (much) at this point.
|Add service API and test service for Yarn Cluster schedulers||Resolved|
|2.||Add YARN extension service to publish Spark events to YARN timeline service||In Progress||Unassigned|
|3.||Add History Service Provider to service application histories from YARN timeline server||Open||Unassigned|