Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2786

Pass Oozie workflow ID and settings to Spark application configuration

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0.0b1, 4.3.1
    • Component/s: action
    • Labels:
      None

      Description

      When using Oozie to launch Spark applications, the Oozie work flow ID and etc do not show up in Spark Application's Environment settings. There is no Spark application ID exposed from Oozie side as well about the Spark applications Oozie launches.

      When looking at applications in Spark History Server, it is hard to figure out which Oozie workflow launched it. This makes debugging the workflow hard. We want to be able to let Spark History server display the Oozie workflow ID in the application's 'Environment' settings.

      For reference, oozie properties that can be passed to Spark are:
      oozie.job.id
      oozie.HadoopAccessorService.created
      oozie.action.spark.setup.hadoop.conf.dir
      oozie.child.mapreduce.job.tags
      oozie.action.id
      oozie.action.rootlogger.log.level

        Attachments

        1. OOZIE-2786-1.patch
          1 kB
          Xiaobin Zheng
        2. OOZIE-2786-2.patch
          4 kB
          Xiaobin Zheng
        3. OOZIE-2786-3.patch
          4 kB
          Xiaobin Zheng

          Issue Links

            Activity

              People

              • Assignee:
                zhengxb2005 Xiaobin Zheng
                Reporter:
                zhengxb2005 Xiaobin Zheng
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: