ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-919

Ephemeral nodes remains in one of ensemble after deliberate SIGKILL

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Duplicate
    • Affects Version/s: 3.3.1
    • Fix Version/s: 3.3.3, 3.4.0
    • Component/s: server
    • Labels:
      None
    • Environment:

      Linux CentOS 5.3 64bit, JDK 1.6.0-22
      SLES 11

      Description

      I was testing stability of Zookeeper ensemble for production deployment. Three node ensemble cluster configuration.
      In a loop, I kill/restart three Zookeeper clients that created one ephemeral node each, and at the same time,
      I killed Java process on one of ensemble (dont' know if it was a leader or not). Then I restarted Zookeeper on the server,

      It turns out that on two zookeeper ensemble servers, all the ephemeral nodes are gone (it should), but on the newly started
      Zookeeper server, the two old ephemeral nodes stayed. The zookeeper didn't restart in standalone mode since new ephemeral
      nodes gets created on all ensemble servers.
      I captured the log.

      2010-11-04 17:48:50,201 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17288:NIOServerCnxn$Factory@250] - Accepted socket connection from /10.25.131.21:11191
      2010-11-04 17:48:50,202 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17288:NIOServerCnxn@776] - Client attempting to establish new session at /10.25.131.21:11191
      2010-11-04 17:48:50,203 - INFO [CommitProcessor:1:NIOServerCnxn@1579] - Established session 0x12c160c31fc000b with negotiated timeout 30000 for client /10.25.131.21:11191
      2010-11-04 17:48:50,206 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17288:NIOServerCnxn@633] - EndOfStreamException: Unable to read additional data from client sessionid 0x12c160c31fc000b, likely client has closed socket
      2010-11-04 17:48:50,207 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17288:NIOServerCnxn@1434] - Closed socket connection for client /10.25.131.21:11191 which had sessionid 0x12c160c31fc000b
      2010-11-04 17:48:50,207 - ERROR [CommitProcessor:1:NIOServerCnxn@444] - Unexpected Exception:
      java.nio.channels.CancelledKeyException
      at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
      at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
      at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:417)
      at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1508)
      at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:367)
      at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)

      1. zk.patch
        2 kB
        Vishal Kher
      2. logs3.tar.gz
        62 kB
        Vishal Kher
      3. logs2.tar.gz
        128 kB
        Vishal Kher
      4. logs.tar.gz
        510 kB
        Vishal Kher

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Chang Song
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development