Affects Version/s: 2.2.0
Fix Version/s: None
Spark 2.2.0 with Mesosphere patches (but the problem exists in main repo)
Parameters passed to driver launching command in Mesos container are escaped using shellEscape function. In
SPARK-18114 additional wrapping in double quotes has been introduced. This cancels out quoting done by shellEscape and makes in unable to run tasks with whitespaces in parameters, as they are interpreted as additional parameters to in-container spark-submit.
This is how parameter passed to in-container spark-submit looks like now:
This is how they look after reverting
SPARK-18114 related commit:
In current version submitting job with such extraJavaOptions causes following error:
SPARK-18114 is the solution to the issue. I can create a pull-request in GitHub. I thought about adding unit tests for that, buth methods generating driver launch command are private.