Details
-
Umbrella
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
1.4.0
-
None
Description
Filing a bug to track different enhancements to the Spark launcher library. Please file sub-tasks for each particular enhancement instead of tagging PRs with this bug's number.
After some discussion in the mailing list, people have requested different enhancements to the library. I'll try to capture those here but feel free to add more in the comments.
- Missing information about the launched Spark application.
Currently the library returns an opaque "Process" object that doesn't have a lot of Spark-related functionality. It would be useful to get at least some information about the underlying process; in the very least the application ID of the actual Spark job. Other useful information could be, for example, the current status of the submitted job.
- Ability to control the underlying application.
The "Process" object only allows you to kill the underlying application. It would be better to have better application-level APIs to try to stop the application more cleanly (e.g. by asking the cluster manager to kill it, or by stopping the SparkContext in client mode).
- Ability to run Spark applications in the same JVM.
This could potentially be done today for cluster mode apps without getting bit by the limitations of SparkContext. In the long run, it would be nice to also support client mode apps.