[CURATOR-315] Reconnect during InterProcessSemaphoreV2.acquire can lead to orphaned node

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 2.10.0
• Fix Version/s: 2.11.0, 3.2.0
• Component/s: Recipes
• Assignee: Jordan Zimmerman
• Reporter: Ulrich Geilmann
• Created: 11/Apr/16 20:55
• Updated: 10/Apr/18 12:10
• Resolved: 08/May/16 18:54

Description

Acquiring and releasing a lease can lead to a leftover node when the client has to reconnect. Subsequent attempts to acquire a lease on the same path fail when max leases is reached due to these nodes.

Here's an excerpt of the client log when that happened:

2016-04-11 10:50:03.634+0000 UTC - INFO org.apache.curator.ConnectionState - Connection attempt unsuccessful after 81663 (greater than max timeout of 60000). Resetting connection and trying again with a new connection. 
2016-04-11 10:50:03.638+0000 UTC - INFO org.apache.zookeeper.ZooKeeper - Session: 0x15404ee16da0015 closed 
2016-04-11 10:50:03.638+0000 UTC - INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=172.16.45.128:2181,172.16.45.192:2181,172.16.45.56:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@34129c78 
2016-04-11 10:50:03.640+0000 UTC - INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.16.45.128/172.16.45.128:2181. Will not attempt to authenticate using SASL (unknown error) 
2016-04-11 10:50:03.640+0000 UTC - INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 172.16.45.128/172.16.45.128:2181, initiating session 
2016-04-11 10:50:03.643+0000 UTC - INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.16.45.128/172.16.45.128:2181, sessionid = 0x25404f0136d0011, negotiated timeout = 40000 
2016-04-11 10:50:03.643+0000 UTC - INFO o.a.c.f.s.ConnectionStateManager - State change: RECONNECTED 
2016-04-11 10:50:03.644+0000 UTC - INFO o.a.c.f.r.l.InterProcessSemaphoreV2 - Sequential path not found: /locking/leases/_c_32448fde-1af3-472f-91ff-ec8a4c4b5034-lease-0000000002 
2016-04-11 10:50:03.644+0000 UTC - INFO org.apache.zookeeper.ClientCnxn - EventThread shut down 
2016-04-11 10:50:06.625+0000 UTC - INFO o.a.c.f.r.l.InterProcessSemaphoreV2 - Lease already released 

Issue Links

• causes: CURATOR-462 InterProcessSemaphoreV2 leaves orphaned lease node if acquiring the semaphore is interrupted at the wrong time