[SPARK-11144] Add SparkLauncher for Spark Streaming, Spark SQL, etc

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.5.1
• Component/s: DStreams, Spark Core, SQL
• Labels: launcher

Environment

Linux x64

• Assignee: Unassigned
• Reporter: Yuhang Chen
• Created: 16/Oct/15 09:12
• Updated: 21/Oct/15 17:18
• Resolved: 21/Oct/15 09:29

Description

Now we hava org.apache.spark.launcher.SparkLauncher to lauch spark as a child process. However, it does not support other libs, such as Spark Streaming and Spark SQL.

What I'm looking for is an utility like spark-submit, with which you can submit any spark lib jobs to all supported resource manager(Standalone, YARN, Mesos, etc) in Java/Scala code.