Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2391

spark-opts value in workflow.xml is not parsed properly

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: trunk
    • Fix Version/s: 4.3.0
    • Component/s: None
    • Labels:

      Description

      When providing spark-opts in workflow.xml for oozie, the "--conf" property values is not parsed properly.

      Example spark opts

      <spark-opts>--conf spark.executor.extraJavaOptions=-Dconfig.url=hdfs:///my.conf -Dmy.env=${env}</spark-opts>
      

      In this case the value for property "spark.executor.extraJavaOptions" is "-Dconfig.url=hdfs:///my.conf -Dmy.env=${env}", but in the oozie code here https://github.com/apache/oozie/blob/38e33811bc3f9d3b1a1a024dcab6af760734b696/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L110 the string is just split by space, which make the "-Dmy.env=${env}" a separate argument to spark and spark fails job as that argument is not valid.

      This doesn't work with enclosing the whole value in quotes too.

      Error from stderr:

      Error: Unrecognized option '-Dmy.env=development"'.
      Run with --help for usage help or --verbose for debug output
      Intercepting System.exit(1)
      Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [1]

        Attachments

        1. OOZIE-2391_addendum.patch
          0.9 kB
          Robert Kanter
        2. OOZIE-2391-001.patch
          1 kB
          Nathan Skone
        3. OOZIE-2391-002.patch
          8 kB
          Peter Cseh
        4. OOZIE-2391-003.patch
          9 kB
          Peter Cseh

          Issue Links

            Activity

              People

              • Assignee:
                gezapeti Peter Cseh
                Reporter:
                pbathala Praveen Bathala
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: