Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2052

Unable to delete a node when the node has no children

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.4.6, 3.5.0
    • Fix Version/s: 3.4.7, 3.5.1, 3.6.0
    • Component/s: server
    • Labels:
      None
    • Environment:

      Red Hat Enterprise Linux 6.1 x86_64, standalone or 3 node ensemble (v3.4.6), 2 Java clients (v3.4.6)

      Description

      We stumbled upon a ZooKeeper bug where a node with no children cannot be removed on our 3 node ZooKeeper ensemble or standalone ZooKeeper on Red Hat Enterprise Linux x86_64 environment. Here is an example scenario/setup:

      o Standalone ZooKeeper or 3 node ensemble (v3.4.6)
      o 2 Java clients (v3.4.6)

      • Client A creates a persistent node (e.g.: /metadata/resources)
      • Client B creates ephemeral nodes under this persistent node

      o Client A attempts to remove the /metadata/resources node via multi op
      delete but fails since there are children
      o Client B's session expired, all the ephemeral nodes are removed
      o Client A attempts to recursively remove /metadata/resources node via
      multi op, this is expected to succeed but got the following exception:
      org.apache.zookeeper.KeeperException$NotEmptyException:
      KeeperErrorCode = Directory not empty

      (Note that Client B is the only client that creates these ephemeral nodes)

      o After this, we use zkCli.sh to inspect the problematic node but the zkCli.sh shows the /metadata/resources node indeed have no children but it will not allow /metadata/resources node to get deleted. (shown below)

      [zk: localhost:2181(CONNECTED) 0] ls /
      [zookeeper, metadata]
      [zk: localhost:2181(CONNECTED) 1] ls /metadata
      [resources]
      [zk: localhost:2181(CONNECTED) 2] get /metadata/resources
      null
      cZxid = 0x3
      ctime = Wed Oct 01 22:04:11 PDT 2014
      mZxid = 0x3
      mtime = Wed Oct 01 22:04:11 PDT 2014
      pZxid = 0x9
      cversion = 2
      dataVersion = 0
      aclVersion = 0
      ephemeralOwner = 0x0
      dataLength = 0
      numChildren = 0
      [zk: localhost:2181(CONNECTED) 3] delete /metadata/resources
      Node not empty: /metadata/resources
      [zk: localhost:2181(CONNECTED) 4] get /metadata/resources
      null
      cZxid = 0x3
      ctime = Wed Oct 01 22:04:11 PDT 2014
      mZxid = 0x3
      mtime = Wed Oct 01 22:04:11 PDT 2014
      pZxid = 0x9
      cversion = 2
      dataVersion = 0
      aclVersion = 0
      ephemeralOwner = 0x0
      dataLength = 0
      numChildren = 0

      o The only ways to remove this node is to either:
      a) Restart the ZooKeeper server
      b) set data to /metadata/resources then followed by a subsequent delete.

        Attachments

        1. ZOOKEEPER-2052-v5-branch34.patch
          18 kB
          Hongchao Deng
        2. ZOOKEEPER-2052-v5.patch
          18 kB
          Hongchao Deng
        3. ZOOKEEPER-2052-v4.patch
          19 kB
          Hongchao Deng
        4. ZOOKEEPER-2052-v3-release.patch
          19 kB
          Hongchao Deng
        5. ZOOKEEPER-2052-v3.patch
          19 kB
          Hongchao Deng
        6. ZOOKEEPER-2052-v2.patch
          20 kB
          Hongchao Deng
        7. ZOOKEEPER-2052.patch
          7 kB
          Yip Ng
        8. ZOOKEEPER-2052.patch
          7 kB
          Yip Ng
        9. ZOOKEEPER-2052.patch
          9 kB
          Yip Ng
        10. zookeeper.log
          30 kB
          Yip Ng
        11. test-jenkins.patch
          12 kB
          Hongchao Deng

          Issue Links

            Activity

              People

              • Assignee:
                hdeng Hongchao Deng
                Reporter:
                yipng Yip Ng
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: