[SPARK-23464] MesosClusterScheduler double-escapes parameters to bash command

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.2.0
• Component/s: Mesos

Environment

Spark 2.2.0 with Mesosphere patches (but the problem exists in main repo)

DC/OS 1.9.5

• Assignee: Unassigned
• Reporter: Marcin Kurczych
• Created: 19/Feb/18 11:26
• Updated: 13/Dec/18 00:41
• Resolved: 13/Dec/18 00:40

Description

Parameters passed to driver launching command in Mesos container are escaped using shellEscape function. In SPARK-18114 additional wrapping in double quotes has been introduced. This cancels out quoting done by shellEscape and makes in unable to run tasks with whitespaces in parameters, as they are interpreted as additional parameters to in-container spark-submit.

This is how parameter passed to in-container spark-submit looks like now:

--conf "spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another""

This is how they look after reverting SPARK-18114 related commit:

--conf spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another"

In current version submitting job with such extraJavaOptions causes following error:

Error: Unrecognized option: -Dbar=another

Usage: spark-submit [options] <app jar | python file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]

Options:
  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
  --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
                              on one of the worker machines inside the cluster ("cluster")
                              (Default: client).
(... further spark-submit help ...)

Reverting SPARK-18114 is the solution to the issue. I can create a pull-request in GitHub. I thought about adding unit tests for that, buth methods generating driver launch command are private.

Issue Links

• duplicates: SPARK-23941 Mesos task failed on specific spark app name
• links to: [Github] Pull Request #20641 (krcz)
• links to: GitHub Pull Request #20641