[SPARK-32839] Make Spark scripts working with the spaces in paths on Windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: Windows
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 10/Sep/20 06:44
• Updated: 12/Dec/22 18:10
• Resolved: 14/Sep/20 04:16

Description

Currently, Spark on Windows does not work with the error messages below:

>>> SparkSession.builder.getOrCreate()
Presence of build for multiple Scala versions detected (C:\...\assembly\target\scala-2.13 and C:\...\assembly\target\scala-2.12).
Remove one of them or, set SPARK_SCALA_VERSION=2.13 in spark-env.cmd.
Visit https://spark.apache.org/docs/latest/configuration.html#environment-variables for more details about setting environment variables in spark-env.cmd.
Either clean one of them or, set SPARK_SCALA_VERSION in spark-env.cmd.

We should handle whitespaces in the cmd scripts at Spark.

Issue Links

• links to: [Github] Pull Request #29706 (HyukjinKwon)