Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
2.2.0
-
None
-
None
-
Spark 2.2.0 with Mesosphere patches (but the problem exists in main repo)
DC/OS 1.9.5
Description
Parameters passed to driver launching command in Mesos container are escaped using shellEscape function. In SPARK-18114 additional wrapping in double quotes has been introduced. This cancels out quoting done by shellEscape and makes in unable to run tasks with whitespaces in parameters, as they are interpreted as additional parameters to in-container spark-submit.
This is how parameter passed to in-container spark-submit looks like now:
--conf "spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another""
This is how they look after reverting SPARK-18114 related commit:
--conf spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another"
In current version submitting job with such extraJavaOptions causes following error:
Error: Unrecognized option: -Dbar=another Usage: spark-submit [options] <app jar | python file> [app arguments] Usage: spark-submit --kill [submission ID] --master [spark://...] Usage: spark-submit --status [submission ID] --master [spark://...] Usage: spark-submit run-example [options] example-class [example args] Options: --master MASTER_URL spark://host:port, mesos://host:port, yarn, or local. --deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or on one of the worker machines inside the cluster ("cluster") (Default: client). (... further spark-submit help ...)
Reverting SPARK-18114 is the solution to the issue. I can create a pull-request in GitHub. I thought about adding unit tests for that, buth methods generating driver launch command are private.
Attachments
Issue Links
- duplicates
-
SPARK-23941 Mesos task failed on specific spark app name
-
- Resolved
-
- links to