Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2052

Unable to delete a node when the node has no children

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.6, 3.5.0
    • 3.4.7, 3.5.1, 3.6.0
    • server
    • None
    • Red Hat Enterprise Linux 6.1 x86_64, standalone or 3 node ensemble (v3.4.6), 2 Java clients (v3.4.6)

    Description

      We stumbled upon a ZooKeeper bug where a node with no children cannot be removed on our 3 node ZooKeeper ensemble or standalone ZooKeeper on Red Hat Enterprise Linux x86_64 environment. Here is an example scenario/setup:

      o Standalone ZooKeeper or 3 node ensemble (v3.4.6)
      o 2 Java clients (v3.4.6)

      • Client A creates a persistent node (e.g.: /metadata/resources)
      • Client B creates ephemeral nodes under this persistent node

      o Client A attempts to remove the /metadata/resources node via multi op
      delete but fails since there are children
      o Client B's session expired, all the ephemeral nodes are removed
      o Client A attempts to recursively remove /metadata/resources node via
      multi op, this is expected to succeed but got the following exception:
      org.apache.zookeeper.KeeperException$NotEmptyException:
      KeeperErrorCode = Directory not empty

      (Note that Client B is the only client that creates these ephemeral nodes)

      o After this, we use zkCli.sh to inspect the problematic node but the zkCli.sh shows the /metadata/resources node indeed have no children but it will not allow /metadata/resources node to get deleted. (shown below)

      [zk: localhost:2181(CONNECTED) 0] ls /
      [zookeeper, metadata]
      [zk: localhost:2181(CONNECTED) 1] ls /metadata
      [resources]
      [zk: localhost:2181(CONNECTED) 2] get /metadata/resources
      null
      cZxid = 0x3
      ctime = Wed Oct 01 22:04:11 PDT 2014
      mZxid = 0x3
      mtime = Wed Oct 01 22:04:11 PDT 2014
      pZxid = 0x9
      cversion = 2
      dataVersion = 0
      aclVersion = 0
      ephemeralOwner = 0x0
      dataLength = 0
      numChildren = 0
      [zk: localhost:2181(CONNECTED) 3] delete /metadata/resources
      Node not empty: /metadata/resources
      [zk: localhost:2181(CONNECTED) 4] get /metadata/resources
      null
      cZxid = 0x3
      ctime = Wed Oct 01 22:04:11 PDT 2014
      mZxid = 0x3
      mtime = Wed Oct 01 22:04:11 PDT 2014
      pZxid = 0x9
      cversion = 2
      dataVersion = 0
      aclVersion = 0
      ephemeralOwner = 0x0
      dataLength = 0
      numChildren = 0

      o The only ways to remove this node is to either:
      a) Restart the ZooKeeper server
      b) set data to /metadata/resources then followed by a subsequent delete.

      Attachments

        1. zookeeper.log
          30 kB
          Yip Ng
        2. ZOOKEEPER-2052.patch
          7 kB
          Yip Ng
        3. ZOOKEEPER-2052.patch
          7 kB
          Yip Ng
        4. ZOOKEEPER-2052.patch
          9 kB
          Yip Ng
        5. test-jenkins.patch
          12 kB
          Hongchao Deng
        6. ZOOKEEPER-2052-v2.patch
          20 kB
          Hongchao Deng
        7. ZOOKEEPER-2052-v3.patch
          19 kB
          Hongchao Deng
        8. ZOOKEEPER-2052-v3-release.patch
          19 kB
          Hongchao Deng
        9. ZOOKEEPER-2052-v4.patch
          19 kB
          Hongchao Deng
        10. ZOOKEEPER-2052-v5-branch34.patch
          18 kB
          Hongchao Deng
        11. ZOOKEEPER-2052-v5.patch
          18 kB
          Hongchao Deng

        Issue Links

          Activity

            People

              hdeng Hongchao Deng
              yipng Yip Ng
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: