Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23464

MesosClusterScheduler double-escapes parameters to bash command

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: Mesos
    • Labels:
      None
    • Environment:

      Spark 2.2.0 with Mesosphere patches (but the problem exists in main repo)

      DC/OS 1.9.5

      Description

      Parameters passed to driver launching command in Mesos container are escaped using shellEscape function. In SPARK-18114 additional wrapping in double quotes has been introduced. This cancels out quoting done by shellEscape and makes in unable to run tasks with whitespaces in parameters, as they are interpreted as additional parameters to in-container spark-submit.

      This is how parameter passed to in-container spark-submit looks like now:

      --conf "spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another""
      

      This is how they look after reverting SPARK-18114 related commit:

      --conf spark.executor.extraJavaOptions="-Dfoo=\"first value\" -Dbar=another"
      

      In current version submitting job with such extraJavaOptions causes following error:

      Error: Unrecognized option: -Dbar=another
      
      Usage: spark-submit [options] <app jar | python file> [app arguments]
      Usage: spark-submit --kill [submission ID] --master [spark://...]
      Usage: spark-submit --status [submission ID] --master [spark://...]
      Usage: spark-submit run-example [options] example-class [example args]
      
      Options:
        --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
        --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
                                    on one of the worker machines inside the cluster ("cluster")
                                    (Default: client).
      (... further spark-submit help ...)
      

      Reverting SPARK-18114 is the solution to the issue. I can create a pull-request in GitHub. I thought about adding unit tests for that, buth methods generating driver launch command are private.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                krcz Marcin Kurczych
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: