[LIVY-217] Add API to access to job status

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.3
• Fix Version/s: 0.9.0
• Component/s: API
• Assignee: Unassigned
• Reporter: gsemet
• Created: 22/Sep/16 07:50
• Updated: 12/Nov/22 22:10

Description

Hello

Following my suggestion on the SparkMagic project (see SparkMagic/#147), I got suggested to submit the following idea for the Livy Project:

• add an API to get job progess (overall and subjobs)
• retrieve the URL of the SparkUI Job information

I also would like to see if it is possible to retreive the javascript code following from the Spark UI

• Event Timeline
• DAG diagram

They provides good information we may want to display, for example, inline in a jupyter notebook (Using SparkMagic).
Like, by pure coincidence, what it done on the Databricks notebook.

Tell me what you think, if it is a good idea, if it fit into existing API (sorry I didnot took the time to get to know the Livy code, only installed on my pc so far), and how it would be the best way to do it.

Thanks