[OOZIE-3113] Retry for ZK lock release

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Satish Saley
• Reporter: Satish Saley
• Created: 09/Nov/17 18:24
• Updated: 02/Jan/18 23:10
• Resolved: 02/Jan/18 23:10

Description

ZK Lock might not get released if release call fails. Oozie should do exponential retry for some time before giving up.

2017-10-25 03:07:45,787  WARN ZKLocksService:523 [pool-12-thread-74] - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[123-123--oozie_saley-C] ACTION[123-123--oozie_saley-C@67] Could not release lock: KeeperErrorCode = ConnectionLoss for /oozie/localhost-oozie/locks/123-123--oozie_saley-C/1234__WRIT__0000000111
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /oozie/localhost-oozie/locks/123-123--oozie_saley-C/1234__WRIT__0000000111
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
        at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:239)
        at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:234)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:230)
        at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:215)
        at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:42)
        at org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:345)
        at org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:123)
        at org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:142)
        at org.apache.oozie.service.ZKLocksService$ZKLockToken.release(ZKLocksService.java:206)
        at org.apache.oozie.command.XCommand.releaseLock(XCommand.java:233)
        at org.apache.oozie.command.XCommand.call(XCommand.java:305)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Issue Links

• links to: Review Board