Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2984

Parse spark-defaults.conf values with spaces without needing the quotes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.3.0
    • Fix Version/s: 5.0.0b1, 4.3.1
    • Component/s: None
    • Labels:
      None
    • Flags:
      Important

      Description

      Oozie requires multiple java opts to be enclosed in quotes. For example:

      spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 -XX:ParallelGCThreads=20"
      

      However the above breaks spark-shell.

      $ ./spark2-shell
      Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 -XX:ParallelGCThreads=20
      Error: Could not create the Java Virtual Machine.
      Error: A fatal exception has occurred. Program will exit.
      

      To fix spark-shell, we have to remove the quotes, which errors out Oozie:

      Error: Unrecognized option '-XX:SurvivorRatio=2'
      ...
      --conf
      spark.executor.extraJavaOptions=-Xmn2703m -Dlog4j.configuration=spark-log4j.properties
      -XX:SurvivorRatio=2
      

      Oozie should be able to parse spark-defaults.conf values with spaces without needing the quotes.

        Attachments

        1. OOZIE-2984.005.patch
          15 kB
          Andras Piros
        2. OOZIE-2984.004.patch
          15 kB
          Andras Piros
        3. OOZIE-2984.003.patch
          13 kB
          Andras Piros
        4. OOZIE-2984.002.patch
          13 kB
          Andras Piros
        5. OOZIE-2984.001.patch
          10 kB
          Andras Piros

          Issue Links

            Activity

              People

              • Assignee:
                andras.piros Andras Piros
                Reporter:
                fahdsiddiqui Fahd Siddiqui
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: