[OOZIE-3458] Can no longer submit a coordinator job which has oozie.libpath and oozie.coord.application.path set

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 5.1.0
• Assignee: Unassigned
• Reporter: Jerry Chabot
• Created: 28/Mar/19 23:45
• Updated: 06/Apr/19 06:05
• Resolved: 29/Mar/19 17:24

Description

This bug is related to https://issues.apache.org/jira/browse/OOZIE-3457. Our Production environment is currently running Amazon EMR 5.19. We want to upgrade to EMR 5.22. Oozie will be upgraded from 5.0.0 to 5.1.0. Our deployment scripts can no longer submit coordinator jobs after upgrading a test environment. Here is a sample post from the oozie client.

POST http://hbase.qa.local:11000/oozie/v2/jobs?action=start

<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>

<property><name>nameNode</name><value>hdfs://hbase.qa.local:8020</value></property>

<property><name>oozie.use.system.libpath</name><value>true</value></property>

<property><name>startTime</name><value>2018-03-28T00:00Z</value></property>

<property><name>queueName</name><value>oozie</value></property>

<property><name>jobTracker</name><value>hbase.qa.local:8032</value></property>

<property><name>oozie.libpath</name><value>hdfs://hbase.qa.local:8020/user/oozie/cmx</value></property>

<property><name>endTime</name><value>2020-01-01:00Z</value></property>

<property><name>applicationConfigFile</name><value>hdfs://hbase.qa.local:8020/usr/share/cmx/conf/app.properties,hdfs://hbase.qa.local:8020/usr/share/cmx/conf/crypto.properties</value></property>

<property><name>frequency</name><value>240</value></property>

<property><name>user.name</name><value>oozie</value></property>

<property><name>oozie.coord.application.path</name><value>hdfs://hbase.qa.local:8020/user/oozie/reporting/workspaceDimToVertica/</value></property>

</configuration>

Error: E0307 : E0307: Runtime error [App directory [null] does not exist and app definition cannot be created because of missing config value [oozie.jobs.api.generated.xml]]

Note, our code specifies oozie.libpath and oozie.coord.application.path. Is this no longer allowed?

I've narrowed down the problem to the checkAndWriteApplicationXMLToHDFS(...) method in V1JobsServlet.java. It is related to the new feature for submitting embedded XML within the job submission. This method is check whether the workflow.xml or coodinator.xml exists in either oozie.wf.application.path, oozie.libpath, oozie.coord.application.path or oozie.bundle.application.path.

First, why is it checking oozie.libpath? That is not an application path.

Second, it stops looking as soon as findAppPathsWithFileNames() returns  a value. Which it will always do when oozie.libpath is set even if an xml doesn't exist there. It does not continue looking for an xml file in oozie.coord.application.path or oozie.bundle.application.path.  I realize only one application path should be specified. IIMO, its the oozie.libpath check that is the problem.

Third, the logic is inefficient. Here are two lines. Why call the same method again with the same params. You already have the result!

if (!findAppPathsWithFileNames(appPath, "workflow.xml").isEmpty())

{ appPathsWithFileNames = findAppPathsWithFileNames(appPath, "workflow.xml"); }

It is critical that we upgrade to EMR 5.22. However, we are blocked until we figure out a way to get the job submitted and completing successfully. Any suggested workarounds would be greatly appreciated. But, I am filing this bug to get it fixed so an upgraded Oozie will make it into a future EMR release. 

Issue Links

• is duplicated by: OOZIE-3415 [core] libpath cannot be used together with coord.application.path and bundle.application.path