[OOZIE-2391] spark-opts value in workflow.xml is not parsed properly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk
• Fix Version/s: 4.3.0
• Labels: newbie
• Assignee: Gézapeti
• Reporter: Praveen Bathala
• Created: 30/Oct/15 23:01
• Updated: 02/Dec/16 21:03
• Resolved: 22/Apr/16 00:48

Description

When providing spark-opts in workflow.xml for oozie, the "--conf" property values is not parsed properly.

Example spark opts

<spark-opts>--conf spark.executor.extraJavaOptions=-Dconfig.url=hdfs:///my.conf -Dmy.env=${env}</spark-opts>

In this case the value for property "spark.executor.extraJavaOptions" is "-Dconfig.url=hdfs:///my.conf -Dmy.env=${env}", but in the oozie code here https://github.com/apache/oozie/blob/38e33811bc3f9d3b1a1a024dcab6af760734b696/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L110 the string is just split by space, which make the "-Dmy.env=${env}" a separate argument to spark and spark fails job as that argument is not valid.

This doesn't work with enclosing the whole value in quotes too.

Error from stderr:

Error: Unrecognized option '-Dmy.env=development"'.
Run with --help for usage help or --verbose for debug output
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [1]

Issue Links

• is duplicated by: OOZIE-2514 Checkstyle violation while doing mvn verify or mvn install