[OOZIE-1821] Oozie java action fails due to AlreadyBeingCreatedException

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: trunk, 3.3.2
• Fix Version/s: 4.1.0
• Component/s: action

Environment

Hadoop 0.20.2-cdh3u5
Oozie server build version: 3.3.2.8

• Assignee: Abhishek Agarwal
• Reporter: Abhishek Agarwal
• Created: 05/May/14 10:32
• Updated: 09/Dec/14 00:45
• Resolved: 03/Jul/14 21:07

Description

2014-04-17 17:13:33,773 WARN ActionStartXCommand:542 - USER[<user_id>] GROUP[-] TOKEN[] APP[<app_name>] JOB[0078484- 140410182733292-oozie-oozi-W] ACTION[0078484-140410182733292-oozie-oozi-W@recordsize] Error starting action [recordsize]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /user/<user-name>/oozie-oozi/0078484-140410182733292-oozie-oozi-W/recordsize--java/action.xml for DFSClient_-1385059579 on client <client ip> because current leaseholder is trying to recreate file.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1407)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1279)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1221)
at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:703)
at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:682)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1439)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1435)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1433)
]

Earlier the java action had failed due to disk quota exceeded exception which is a transient error. when oozie retried to run the action, later retrials resulted in above exception.

In org.apache.oozie.action.hadoop,LauncherMapper, if writing action xml fails due to an error, output stream should be closed gracefully.

OutputStream os = fs.create(new Path(actionDir, ACTION_CONF_XML));
actionConf.writeXml(os);
os.close();

Issue Links

• is duplicated by: OOZIE-1822 Oozie java action fails due to AlreadyBeingCreatedException