Oozie
  1. Oozie
  2. OOZIE-1580

EL variables don't get resolved in configurations imported from a <job-xml>

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1.0
    • Component/s: None
    • Labels:
      None

      Description

      If you use <job-xml> to include a file that includes an EL variable, it doesn't get resolved.

      For example:

      foo.xml
      <configuration>
         <property>
            <name>some.property</name>
            <value>${someVariable}</value>
         </property>
      </configuration>
      
      job.propertiesl
      ...
      someVariable=bar
      

      Then in the submitted job, some.property will be equal to "${someVariable}" when we would like it to be "bar".

        Issue Links

          Activity

          Hide
          Robert Kanter added a comment -

          Thanks Bowen. Committed to trunk!

          Show
          Robert Kanter added a comment - Thanks Bowen. Committed to trunk!
          Hide
          Robert Kanter added a comment -

          Talked to Bowen offline, the error I was running into is a different issue and is being address by OOZIE-1600.

          +1

          Show
          Robert Kanter added a comment - Talked to Bowen offline, the error I was running into is a different issue and is being address by OOZIE-1600 . +1
          Hide
          Robert Kanter added a comment -

          It's just the contents of the <configuration> section from the example:

          <configuration>
              <property>
                  <name>mapred.job.queue.name</name>
                  <value>${queueName}</value>
              </property>
              <property>
                  <name>mapred.mapper.class</name>
                  <value>org.apache.oozie.example.SampleMapper</value>
              </property>
              <property>
                  <name>mapred.reducer.class</name>
                  <value>org.apache.oozie.example.SampleReducer</value>
              </property>
              <property>
                  <name>mapred.map.tasks</name>
                  <value>1</value>
              </property>
              <property>
                  <name>mapred.input.dir</name>
                  <value>/user/${wf:user()}/${examplesRoot}/input-data/text</value>
              </property>
              <property>
                  <name>mapred.output.dir</name>
                  <value>/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}</value>
              </property>
          </configuration>
          
          Show
          Robert Kanter added a comment - It's just the contents of the <configuration> section from the example: <configuration> <property> <name> mapred.job.queue.name </name> <value> ${queueName} </value> </property> <property> <name> mapred.mapper.class </name> <value> org.apache.oozie.example.SampleMapper </value> </property> <property> <name> mapred.reducer.class </name> <value> org.apache.oozie.example.SampleReducer </value> </property> <property> <name> mapred.map.tasks </name> <value> 1 </value> </property> <property> <name> mapred.input.dir </name> <value> /user/${wf:user()}/${examplesRoot}/input-data/text </value> </property> <property> <name> mapred.output.dir </name> <value> /user/${wf:user()}/${examplesRoot}/output-data/${outputDir} </value> </property> </configuration>
          Hide
          Bowen Zhang added a comment -

          Robert Kanter, Can you show me the foo.xml?

          Show
          Bowen Zhang added a comment - Robert Kanter , Can you show me the foo.xml?
          Hide
          Robert Kanter added a comment -

          Ah, I missed that its calling JavaActionExecutor; make sense.

          I tried editing the mapreduce example by moving its <configuration> properties to a separate xml file and including that with <job-xml:

                  <map-reduce>
                      <job-tracker>${jobTracker}</job-tracker>
                      <name-node>${nameNode}</name-node>
                      <prepare>
                          <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/>
                      </prepare>
                      <job-xml>foo.xml</job-xml>
                  </map-reduce>
          

          The MR job succeeded, but Oozie had an IllegalArgumentException somewhere:

          $ oozie job -info 0000000-131028144305717-oozie-rkan-W@mr-node
          ID : 0000000-131028144305717-oozie-rkan-W@mr-node
          ------------------------------------------------------------------------------------------------------------------------------------
          Console URL       : http://localhost:50030/jobdetails.jsp?jobid=job_201310281439_0001
          Error Code        : IllegalArgumentException
          Error Message     : IllegalArgumentException: element cannot be null
          External ID       : job_201310281439_0001
          External Status   : SUCCEEDED
          Name              : mr-node
          Retries           : 0
          Tracker URI       : localhost:8021
          Type              : map-reduce
          Started           : 2013-10-28 21:44 GMT
          Status            : ERROR
          Ended             : -
          ------------------------------------------------------------------------------------------------------------------------------------
          

          You can see that the "external status" is "SUCCEEDED" (and I checked the MR job); but the action itself was "ERROR". I looked at the Oozie log, but all I see in there is this:

          2013-10-28 14:44:50,525  WARN ActionEndXCommand:542 - SERVER[rkanter-MBP.local] USER[rkanter] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-131028144305717-oozie-rkan-W] ACTION[0000000-131028144305717-oozie-rkan-W@mr-node] Error ending action [mr-node]. ErrorType [ERROR], ErrorCode [IllegalArgumentException], Message [IllegalArgumentException: element cannot be null]
          

          Can you take a look?

          Show
          Robert Kanter added a comment - Ah, I missed that its calling JavaActionExecutor ; make sense. I tried editing the mapreduce example by moving its <configuration> properties to a separate xml file and including that with <job-xml : <map-reduce> <job-tracker> ${jobTracker} </job-tracker> <name-node> ${nameNode} </name-node> <prepare> <delete path= "${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}" /> </prepare> <job-xml> foo.xml </job-xml> </map-reduce> The MR job succeeded, but Oozie had an IllegalArgumentException somewhere: $ oozie job -info 0000000-131028144305717-oozie-rkan-W@mr-node ID : 0000000-131028144305717-oozie-rkan-W@mr-node ------------------------------------------------------------------------------------------------------------------------------------ Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201310281439_0001 Error Code : IllegalArgumentException Error Message : IllegalArgumentException: element cannot be null External ID : job_201310281439_0001 External Status : SUCCEEDED Name : mr-node Retries : 0 Tracker URI : localhost:8021 Type : map-reduce Started : 2013-10-28 21:44 GMT Status : ERROR Ended : - ------------------------------------------------------------------------------------------------------------------------------------ You can see that the "external status" is "SUCCEEDED" (and I checked the MR job); but the action itself was "ERROR". I looked at the Oozie log, but all I see in there is this: 2013-10-28 14:44:50,525 WARN ActionEndXCommand:542 - SERVER[rkanter-MBP.local] USER[rkanter] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-131028144305717-oozie-rkan-W] ACTION[0000000-131028144305717-oozie-rkan-W@mr-node] Error ending action [mr-node]. ErrorType [ERROR], ErrorCode [IllegalArgumentException], Message [IllegalArgumentException: element cannot be null] Can you take a look?
          Hide
          Bowen Zhang added a comment -

          FsActionExecutor does call the method parseJobXmlAndConfiguration from doOperations(). And all the other subclasses of actionExecutor don't invoke any checks for "job-xml" tab because (i guess) it won't make sense to do it. This is either by feature design or an existing bug in the code base which in either case should be outside the scope of this ticket.

          Show
          Bowen Zhang added a comment - FsActionExecutor does call the method parseJobXmlAndConfiguration from doOperations(). And all the other subclasses of actionExecutor don't invoke any checks for "job-xml" tab because (i guess) it won't make sense to do it. This is either by feature design or an existing bug in the code base which in either case should be outside the scope of this ticket.
          Hide
          Robert Kanter added a comment -

          The changes will only affect actions that subclass the JavaActionExecutor; but some action executors, such as the FsActionExecutor do not so they won't resolve the variables in their <job-xml> if they have one. I think we should add a function to ActionExecutor that can resolve the <job-xml> variables and then any subclasses (i.e. JavaActionExecutor, FsActionExecutor), can simply call it.

          Show
          Robert Kanter added a comment - The changes will only affect actions that subclass the JavaActionExecutor ; but some action executors, such as the FsActionExecutor do not so they won't resolve the variables in their <job-xml> if they have one. I think we should add a function to ActionExecutor that can resolve the <job-xml> variables and then any subclasses (i.e. JavaActionExecutor , FsActionExecutor ), can simply call it.
          Hide
          Bowen Zhang added a comment -

          test failure unrelated.

          Show
          Bowen Zhang added a comment - test failure unrelated.
          Hide
          Hadoop QA added a comment -

          Testing JIRA OOZIE-1580

          Cleaning local svn workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          +1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . +1 the patch does adds/modifies 2 testcase(s)
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS
          . Tests run: 1348
          . Tests failed: 1
          . Tests errors: 0

          . The patch failed the following testcases:

          . testTimeoutWithException(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)

          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/866/

          Show
          Hadoop QA added a comment - Testing JIRA OOZIE-1580 Cleaning local svn workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . +1 the patch does adds/modifies 2 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS . Tests run: 1348 . Tests failed: 1 . Tests errors: 0 . The patch failed the following testcases: . testTimeoutWithException(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand) +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/866/
          Hide
          Robert Kanter added a comment -

          Ya, including configs in a <job-xml> should have the same behavior as if you included them directly in the <configuration> section.

          Show
          Robert Kanter added a comment - Ya, including configs in a <job-xml> should have the same behavior as if you included them directly in the <configuration> section.
          Hide
          Bowen Zhang added a comment -

          Do we want to include function variable in the foo.xml file like $

          {wf:user()}

          in addition to pure variable substitution like $

          {someVariable}

          ?

          Show
          Bowen Zhang added a comment - Do we want to include function variable in the foo.xml file like $ {wf:user()} in addition to pure variable substitution like $ {someVariable} ?

            People

            • Assignee:
              Bowen Zhang
              Reporter:
              Robert Kanter
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development