Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1958

address duplication of env variables in oozie.launcher.yarn.app.mapreduce.am.env when running with uber mode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Hadoop 2.X has the weird behavior to translate yarn.app.mapreduce.am.env,
      JAVA_HOME=/home/gs/java/jdk64/current,JAVA_HOME=/home/gs/java/jdk64/current
      to
      JAVA_HOME=/home/gs/java/jdk64/current:JAVA_HOME=/home/gs/java/jdk64/current

      due to this, some of env variables are not taken in effect properly.

      this problem happens when running launcher with uber mode in Hadoop 2.X,
      since oozie copies mapreduce.map.env to yarn.app.mapreduce.am.env, like following case

      <property>
            <name>oozie.launcher.mapreduce.map.env</name>
            <value>JAVA_HOME=/home/gs/java/jdk64/current</value>
      </property>
      <property>
            <name>oozie.launcher.yarn.app.mapreduce.am.env</name>
            <value>JAVA_HOME=/home/gs/java/jdk64/current</value>
      </property>
      

      this patch is to do
      1) Deduplication before appending map.env to am.env
      2) If there is a conflict, turn of uber mode AM.

        Attachments

        1. OOZIE-1958-v3.patch
          13 kB
          Ryota Egashira

          Issue Links

            Activity

              People

              • Assignee:
                egashira Ryota Egashira
                Reporter:
                egashira Ryota Egashira
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: