[SPARK-4924] Factor out code to launch Spark applications into a separate library

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.4.0
• Component/s: Spark Core
• Target Version/s: 1.4.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 23/Dec/14 00:06
• Updated: 16/May/16 14:04
• Resolved: 11/Mar/15 08:03

Description

One of the questions we run into rather commonly is "how to start a Spark application from my Java/Scala program?". There currently isn't a good answer to that:

• Instantiating SparkContext has limitations (e.g., you can only have one active context at the moment, plus you lose the ability to submit apps in cluster mode)
• Calling SparkSubmit directly is doable but you lose a lot of the logic handled by the shell scripts
• Calling the shell script directly is doable, but sort of ugly from an API point of view.

I think it would be nice to have a small library that handles that for users. On top of that, this library could be used by Spark itself to replace a lot of the code in the current shell scripts, which have a lot of duplication.

Issue Links

• is duplicated by: SPARK-3733 Support for programmatically submitting Spark jobs
• relates to: SPARK-6047 pyspark - class loading on driver failing with --jars and --packages
• links to: [Github] Pull Request #3916 (vanzin)