[ZOOKEEPER-2052] Unable to delete a node when the node has no children

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.6, 3.5.0
• Fix Version/s: 3.4.7, 3.5.1, 3.6.0
• Component/s: server

Environment

Red Hat Enterprise Linux 6.1 x86_64, standalone or 3 node ensemble (v3.4.6), 2 Java clients (v3.4.6)

• Assignee: Hongchao Deng
• Reporter: Yip Ng
• Created: 02/Oct/14 07:08
• Updated: 28/Oct/14 11:10
• Resolved: 28/Oct/14 04:41

Description

We stumbled upon a ZooKeeper bug where a node with no children cannot be removed on our 3 node ZooKeeper ensemble or standalone ZooKeeper on Red Hat Enterprise Linux x86_64 environment. Here is an example scenario/setup:

o Standalone ZooKeeper or 3 node ensemble (v3.4.6)
o 2 Java clients (v3.4.6)

• Client A creates a persistent node (e.g.: /metadata/resources)
• Client B creates ephemeral nodes under this persistent node

o Client A attempts to remove the /metadata/resources node via multi op
delete but fails since there are children
o Client B's session expired, all the ephemeral nodes are removed
o Client A attempts to recursively remove /metadata/resources node via
multi op, this is expected to succeed but got the following exception:
org.apache.zookeeper.KeeperException$NotEmptyException:
KeeperErrorCode = Directory not empty

(Note that Client B is the only client that creates these ephemeral nodes)

o After this, we use zkCli.sh to inspect the problematic node but the zkCli.sh shows the /metadata/resources node indeed have no children but it will not allow /metadata/resources node to get deleted. (shown below)

[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper, metadata]
[zk: localhost:2181(CONNECTED) 1] ls /metadata
[resources]
[zk: localhost:2181(CONNECTED) 2] get /metadata/resources
null
cZxid = 0x3
ctime = Wed Oct 01 22:04:11 PDT 2014
mZxid = 0x3
mtime = Wed Oct 01 22:04:11 PDT 2014
pZxid = 0x9
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: localhost:2181(CONNECTED) 3] delete /metadata/resources
Node not empty: /metadata/resources
[zk: localhost:2181(CONNECTED) 4] get /metadata/resources
null
cZxid = 0x3
ctime = Wed Oct 01 22:04:11 PDT 2014
mZxid = 0x3
mtime = Wed Oct 01 22:04:11 PDT 2014
pZxid = 0x9
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0

o The only ways to remove this node is to either:
a) Restart the ZooKeeper server
b) set data to /metadata/resources then followed by a subsequent delete.

Issue Links

• is related to: ZOOKEEPER-1424 ZooKeeper will not allow a client to delete a tree when it should allow it