[SPARK-3644] REST API for Spark application info (jobs / stages / tasks / storage info)

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Spark Core, Web UI
• Assignee: Imran Rashid
• Reporter: Josh Rosen
• Created: 22/Sep/14 18:34
• Updated: 30/Jul/15 10:41
• Resolved: 19/May/15 18:40

Description

This JIRA is a forum to draft a design proposal for a REST interface for accessing information about Spark applications, such as job / stage / task / storage status.

There have been a number of proposals to serve JSON representations of the information displayed in Spark's web UI. Given that we might redesign the pages of the web UI (and possibly re-implement the UI as a client of a REST API), the API endpoints and their responses should be independent of what we choose to display on particular web UI pages / layouts.

Let's start a discussion of what a good REST API would look like from first-principles. We can discuss what urls / endpoints expose access to data, how our JSON responses will be formatted, how fields will be named, how the API will be documented and tested, etc.

Some links for inspiration:

https://developer.github.com/v3/
http://developer.netflix.com/docs/REST_API_Reference
https://helloreverb.com/developers/swagger

Issue Links

• is related to: SPARK-2321 Design a proper progress reporting & event listener API
• is related to: SPARK-3024 CLI interface to Driver
• relates to: SPARK-3454 Expose JSON representation of data shown in WebUI
• relates to: SPARK-907 Add JSON endpoints to SparkUI