Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-832

Invalid session id causes infinite loop during automatic reconnect

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.4.5, 3.5.0, 3.4.11
    • Fix Version/s: None
    • Component/s: server
    • Labels:
      None
    • Environment:

      All

      Description

      Steps to reproduce:

      1.) Connect to a standalone server using the Java client.
      2.) Stop the server.
      3.) Delete the contents of the data directory (i.e. the persisted session data).
      4.) Start the server.

      The client now automatically tries to reconnect but the server refuses the connection because the session id is invalid. The client and server are now in an infinite loop of attempted and rejected connections. While this situation represents a catastrophic failure and the current behavior is not incorrect, it appears that there is no way to detect this situation on the client and therefore no way to recover.

      The suggested improvement is to send an event to the default watcher indicating that the current state is "session invalid", similar to how the "session expired" state is handled.

      Server log output (repeats indefinitely):
      2010-08-05 11:48:08,283 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@250] - Accepted socket connection from /127.0.0.1:63292
      2010-08-05 11:48:08,284 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@751] - Refusing session request for client /127.0.0.1:63292 as it has seen zxid 0x44 our last zxid is 0x0 client must try another server
      2010-08-05 11:48:08,284 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1434] - Closed socket connection for client /127.0.0.1:63292 (no session established for client)

      Client log output (repeats indefinitely):
      11:47:17 org.apache.zookeeper.ClientCnxn startConnect INFO line 1000 - Opening socket connection to server localhost/127.0.0.1:2181
      11:47:17 org.apache.zookeeper.ClientCnxn run WARN line 1120 - Session 0x12a3ae4e893000a for server null, unexpected error, closing socket connection and attempting reconnect
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
      11:47:17 org.apache.zookeeper.ClientCnxn cleanup DEBUG line 1167 - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1164)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1129)
      11:47:17 org.apache.zookeeper.ClientCnxn cleanup DEBUG line 1174 - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1171)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1129)

        Attachments

        1. ZOOKEEPER-832.patch
          16 kB
          Mohammad Arshad
        2. ZOOKEEPER-832.patch
          6 kB
          Germán Blanco
        3. ZOOKEEPER-832.patch
          6 kB
          Germán Blanco
        4. ZOOKEEPER-832.patch
          6 kB
          Germán Blanco
        5. ZOOKEEPER-832.patch
          6 kB
          Germán Blanco
        6. ZOOKEEPER-832.patch
          6 kB
          Germán Blanco
        7. ZOOKEEPER-832.patch
          6 kB
          Germán Blanco
        8. ZOOKEEPER-832.patch
          6 kB
          Germán Blanco
        9. ZOOKEEPER-832.patch
          8 kB
          Germán Blanco
        10. ZOOKEEPER-832.patch
          0.8 kB
          Germán Blanco

          Issue Links

            Activity

              People

              • Assignee:
                arshad.mohammad Mohammad Arshad
                Reporter:
                rholmes Ryan Holmes
              • Votes:
                13 Vote for this issue
                Watchers:
                40 Start watching this issue

                Dates

                • Created:
                  Updated: