[CURATOR-268] Curator client doesn't behave well when it loses connection: CRUD operation fail

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.1, 2.9.0
• Fix Version/s: 3.0.0
• Component/s: Framework
• Assignee: Jordan Zimmerman
• Reporter: Benjamin Jaton
• Created: 01/Oct/15 22:54
• Updated: 09/Aug/16 11:13
• Resolved: 06/Oct/15 01:43

Description

I am doing a basic stress test :

• a TestingServer that keeps restarting in one thread
• a Curator client that keeps creating/deleting a node in another

After a few seconds, ether the create or the delete fail:

Thu Oct 01 15:35:10 PDT 2015 - Node /test has successfully been removed.
Thu Oct 01 15:35:10 PDT 2015 - Recreating /test
Thu Oct 01 15:35:10 PDT 2015 - Restarting server...
Thu Oct 01 15:35:14 PDT 2015 - Restarting server...
Thu Oct 01 15:35:15 PDT 2015 - ERROR: node should be removed.
Thu Oct 01 15:35:15 PDT 2015 - Data of node is: test
Node has been mysteriously created...
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /test
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1209)
	at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:720)
	at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:703)
	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
	at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:700)
	at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:477)
	at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:467)
	at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
	at TestCuratorSaveConnLoss$3.run(TestCuratorSaveConnLoss.java:87)

The create shouldn't fail, we're within 5 seconds of the create and the connection timeout is 10 secs.

This is surprising as this basic scenario is - AFAIK - the reason Curator exists in the first place.

Issue Links

• is related to: ZOOKEEPER-2288 During shutdown, server may fail to ack completed transactions to clients.
• relates to: CURATOR-341 create().orSetData() retries indefinitely if node exists