[OOZIE-2984] Parse spark-defaults.conf values with spaces without needing the quotes

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.3.0
• Fix Version/s: 5.0.0b1, 4.3.1
• Flags: Important
• Assignee: Andras Piros
• Reporter: Fahd Siddiqui
• Created: 11/Jul/17 05:58
• Updated: 17/Aug/18 12:28
• Resolved: 28/Jul/17 11:34

Description

Oozie requires multiple java opts to be enclosed in quotes. For example:

spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 -XX:ParallelGCThreads=20"

However the above breaks spark-shell.

$ ./spark2-shell
Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 -XX:ParallelGCThreads=20
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

To fix spark-shell, we have to remove the quotes, which errors out Oozie:

Error: Unrecognized option '-XX:SurvivorRatio=2'
...
--conf
spark.executor.extraJavaOptions=-Xmn2703m -Dlog4j.configuration=spark-log4j.properties
-XX:SurvivorRatio=2

Oozie should be able to parse spark-defaults.conf values with spaces without needing the quotes.

Issue Links

• is duplicated by: OOZIE-3050 Parse Error if the spark job has the spark.executor.extraClassPath confguration.
• relates to: OOZIE-3330 [spark-action] Remove double quotes inside plain option values
• links to: ReviewBoard link