[SPARK-35124] Local mode fails to start cluster due to configuration value escape issue

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: Java API
• Assignee: Unassigned
• Reporter: Malthe Borch
• Created: 18/Apr/21 06:07
• Updated: 19/Apr/21 18:06

Description

At least on Windows and perhaps also other systems, running Spark in local mode fails to start a cluster when a configuration key contains a value with "&" (ampersand) in it.

This happens during the "getOrCreate()" call from a Spark session builder.

The reason seems to be incorrect or insufficient escaping since on Windows, the attempt to start a Spark process actually ends up running multiple commands for each occurrence of the "&" character.

On Windows specifically, the correct way to escape "&" would be "^&", but I have not been able yet to figure out exactly where the process is started and how the configuration is passed.

Issue Links

• links to: Pull request