Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3228

[Spark action] Can't load properties from spark-defaults.conf

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 5.0.0, 4.3.1
    • 5.1.0
    • action
    • None

    Description

      When I create a Oozie workflow to launch a spark action, the spark job can't load the configured properties in spark-defaults.conf. I've configured each NodeManager as the Spark gateway role, so spark-defaults.conf is generated in /etc/spark/conf/ on each worker node.

      In spark-defaults.conf some configuration I've set into.

      spark.executor.extraClassPath=/etc/hbase/conf:/etc/hive/conf
spark.driver.extraClassPath=/etc/hbase/conf:/etc/hive/conf

      But in the Oozie spark job, they're not loaded automatically.

      --conf spark.executor.extraClassPath=$PWD/*
--conf spark.driver.extraClassPath=$PWD/*

      Attachments

        1. OOZIE-3228.001.patch
          8 kB
          Andras Piros
        2. OOZIE-3228.002.patch
          9 kB
          Andras Piros
        3. OOZIE-3228.003.patch
          10 kB
          Andras Piros
        4. OOZIE-3228.004.patch
          10 kB
          Andras Piros
        5. OOZIE-3228.005.patch
          12 kB
          Andras Piros
        6. OOZIE-3228.amend.001.patch
          5 kB
          Andras Piros

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. OOZIE-3262 [Spark action] Improve TestSparkArgsExtractor

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. OOZIE-3112 SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. OOZIE-2923 Improve Spark options parsing

          • Major - Major loss of function.
          • Closed
          links to

          Web Link ReviewBoard link

          Activity

            People

              andras.piros Andras Piros
              Tang Yan Tang Yan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: