[SPARK-2722] Mechanism for escaping spark configs is not consistent

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.0.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Andrew Or
• Created: 29/Jul/14 01:10
• Updated: 08/Oct/16 09:38
• Resolved: 08/Oct/16 09:38

Description

Currently, you can specify a spark config in spark-defaults.conf as follows:

spark.magic "Mr. Johnson"

and this will preserve the double quotes as part of the string. Naturally, if you want to do the equivalent in spark.*.extraJavaOptions, you would use the following:

spark.executor.extraJavaOptions "-Dmagic=\"Mr. Johnson\""

However, this fails because the backslashes go away and it tries to interpret "Johnson" as the main class argument. Instead, you have to do the following:

spark.executor.extraJavaOptions "-Dmagic=\\\"Mr. Johnson\\\""

which is not super intuitive.

Note that this only applies to standalone mode. In YARN it's not even possible to use quoted strings in config values (SPARK-2718).

Issue Links

• is duplicated by: SPARK-2914 spark.*.extraJavaOptions are evaluated too many times
• is related to: SPARK-2718 YARN does not handle spark configs with quotes or backslashes
• relates to: SPARK-3620 Refactor config option handling code for spark-submit