ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1412

java client watches inconsistently triggered on reconnect

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.3.3, 3.3.4, 3.4.0, 3.4.1, 3.4.2, 3.4.3
    • Fix Version/s: 3.3.5, 3.4.4, 3.5.0
    • Component/s: server
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I've observed an inconsistent behavior in java client watches. The inconsistency relates to the behavior after the client reconnects to the zookeeper ensemble.

      After the client reconnects to the ensemble only those watches should trigger which should have been triggered also if the connections was not lost. This means if I watch for changes in node /foo and there is no change there than my watch should not be triggered on reconnecting to the ensemble.
      This is not always the case in the java client.

      I've debugged the issues and I could locate the case when the watch is always triggered on reconnect. This is consistently happening if I connect to a follower in the ensemble and I don't do any operation which goes through the leader.
      Looking at the code I see that the client stores the lastzxid and sends that with its request. This is 0 on startup and will be updated everytime from the server replies. This lastzxid is also sent to the server after reconnect together with watches. The server decides which watch to trigger based on this lastzxid probably because that should mean the last known state of the client. If this lastzxid is 0 than all the watches are triggered.
      I've checked why is this lastzxid 0. I thought it shouldn't be since there was already a request to the server to set the watch and in the reply the server could have sent back the zxid but it turns out that it sends just 0. Looking at the server code I see that for requests which doesn't go through the leader the follower server just sends back the same zxid that the client sent.

      1. ZOOKEEPER-1412_br33.patch
        6 kB
        Patrick Hunt
      2. ZOOKEEPER-1412_br34.patch
        8 kB
        Patrick Hunt
      3. ZOOKEEPER-1412_trunk.patch
        8 kB
        Patrick Hunt
      4. ZOOKEEPER-1412_br33.patch
        6 kB
        Patrick Hunt
      5. ZOOKEEPER-1412_br34.patch
        8 kB
        Patrick Hunt
      6. ZOOKEEPER-1412_trunk.patch
        8 kB
        Patrick Hunt

        Issue Links

          Activity

          Botond Hejj created issue -
          Patrick Hunt made changes -
          Field Original Value New Value
          Assignee Patrick Hunt [ phunt ]
          Patrick Hunt made changes -
          Fix Version/s 3.5.0 [ 12316644 ]
          Patrick Hunt made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Component/s server [ 12312382 ]
          Component/s java client [ 12312381 ]
          Patrick Hunt made changes -
          Attachment ZOOKEEPER-1412_br33.patch [ 12518088 ]
          Attachment ZOOKEEPER-1412_br34.patch [ 12518089 ]
          Patrick Hunt made changes -
          Attachment ZOOKEEPER-1412_trunk.patch [ 12518091 ]
          Patrick Hunt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Patrick Hunt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Patrick Hunt made changes -
          Attachment ZOOKEEPER-1412_br33.patch [ 12518109 ]
          Attachment ZOOKEEPER-1412_br34.patch [ 12518110 ]
          Patrick Hunt made changes -
          Attachment ZOOKEEPER-1412_trunk.patch [ 12518111 ]
          Patrick Hunt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Patrick Hunt made changes -
          Link This issue relates to ZOOKEEPER-1417 [ ZOOKEEPER-1417 ]
          Patrick Hunt made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Patrick Hunt made changes -
          Hadoop Flags Reviewed [ 10343 ]

            People

            • Assignee:
              Patrick Hunt
              Reporter:
              Botond Hejj
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development