Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2974

Change TestLiteWorkflowAppParser so that it works with Hadoop 3

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0.0b1
    • Component/s: None
    • Labels:
      None

      Description

      TestLiteWorkflowAppParser uses a cleanupXML function to remove certain elements from a generated configuration XML when comparing with an expected value (testParserGlobal() ):

       private String cleanupXml(String xml) {
              xml = xml.replaceAll(" xmlns=?(\"|\')(\"|\')", "");
              xml = xml.replaceAll("\\s*<source>.*</source>", "");    // remove the <source> added by Hadoop 2
              return xml;
          }
      

      HADOOP-14216 introduced a new "final" tag in the configuration that is set when hadoop 3 alpha 3 is used. See https://github.com/apache/hadoop/blob/branch-3.0.0-alpha3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2952

      If we run the the test TestLiteWorkflowAppParser#testParserGlobal, it currently fails with:

      Failed tests:   testParserGlobal(org.apache.oozie.workflow.lite.TestLiteWorkflowAppParser): null expected:<...(..)
      

      generated config that is returned by app.getNode("d").getConf() is as follows:

      <map-reducexmlns="uri:oozie:workflow:0.4">
      <prepare>
      <deletepath="/tmp"/>
      <mkdirpath="/tmp"/>
      </prepare>
      <streaming>
      <mapper>/mycat.sh</mapper>
      <reducer>/mywc.sh</reducer>
      </streaming>
      <file>/tmp</file>
      <archive>/tmp</archive>
      <name-node>bar</name-node>
      <job-tracker>${foo}</job-tracker>
      <configuration>
      <property>
      <name>b</name>
      <value>B</value>
      <final>false</final>
      </property>
      <property>
      <name>a</name>
      <value>A</value>
      <final>false</final>
      </property>
      </configuration>
      </map-reduce>
      

      Tests pass, if we remove the final element. For example we can the following to cleanupXml()

      xml = xml.replaceAll("\\s*<final>.*</final>", "");    // remove the <final> added by Hadoop 3
      

        Attachments

        1. OOZIE-2974-02.patch
          0.9 kB
          Attila Sasvári
        2. OOZIE-2974-01.patch
          0.9 kB
          Attila Sasvári

          Issue Links

            Activity

              People

              • Assignee:
                asasvari Attila Sasvári
                Reporter:
                asasvari Attila Sasvári
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: