[HIVE-16484] Investigate SparkLauncher for HoS as alternative to bin/spark-submit - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Done
Affects Version/s: None
Fix Version/s: None
Component/s: Spark
Labels:
None

Description

The SparkClientImpl#startDriver currently looks for the SPARK_HOME directory and invokes the bin/spark-submit script, which spawns a separate process to run the Spark application.

SparkLauncher was added in ~~SPARK-4924~~ and is a programatic way to launch Spark applications.

I see a few advantages:

No need to spawn a separate process to launch a HoS --> lower startup time
Simplifies the code in SparkClientImpl --> easier to debug
SparkLauncher#startApplication returns a SparkAppHandle which contains some useful utilities for querying the state of the Spark job
- It also allows the launcher to specify a list of job listeners

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HIVE-16484.1.patch
20/Apr/17 20:03
32 kB
Sahil Takiar
HIVE-16484.10.patch
05/Jan/18 21:51
51 kB
Sahil Takiar
HIVE-16484.2.patch
21/Apr/17 19:35
32 kB
Sahil Takiar
HIVE-16484.3.patch
21/Apr/17 23:33
33 kB
Sahil Takiar
HIVE-16484.4.patch
24/Apr/17 21:07
49 kB
Sahil Takiar
HIVE-16484.5.patch
26/Apr/17 17:07
49 kB
Sahil Takiar
HIVE-16484.6.patch
28/Apr/17 05:42
49 kB
Sahil Takiar
HIVE-16484.7.patch
28/Apr/17 18:13
52 kB
Sahil Takiar
HIVE-16484.8.patch
02/Jan/18 22:48
48 kB
Sahil Takiar
HIVE-16484.9.patch
03/Jan/18 21:33
49 kB
Sahil Takiar

Issue Links

is related to

HIVE-18533 Add option to use InProcessLauncher to submit spark jobs

Closed

links to

Activity

People

Assignee:: Sahil Takiar

Reporter:: Sahil Takiar

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 19/Apr/17 22:22

Updated:: 25/Jan/18 00:52

Resolved:: 11/Jan/18 00:17