Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-1277

servers stop serving when lower 32bits of zxid roll over

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.3.3
    • 3.3.5, 3.4.4, 3.5.0
    • server
    • None
    • Reviewed
    • Hide
      Workaround: there is a simple workaround for this issue. Force a leader re-election before the lower 32bits reach 0xffffffff

      Most users won't even see this given the number of writes on a typical installation - say you are doing 500 writes/second, you'd see this after ~3 months if the quorum is stable, any changes (such as upgrading the server software) would cause the xid to be reset, thereby staving off this issue for another period.
      Show
      Workaround: there is a simple workaround for this issue. Force a leader re-election before the lower 32bits reach 0xffffffff Most users won't even see this given the number of writes on a typical installation - say you are doing 500 writes/second, you'd see this after ~3 months if the quorum is stable, any changes (such as upgrading the server software) would cause the xid to be reset, thereby staving off this issue for another period.

    Description

      When the lower 32bits of a zxid "roll over" (zxid is a 64 bit number, however the upper 32 are considered the epoch number) the epoch number (upper 32 bits) are incremented and the lower 32 start at 0 again.

      This should work fine, however in the current 3.3 branch the followers see this as a NEWLEADER message, which it's not, and effectively stop serving clients. Attached clients seem to eventually time out given that heartbeats (or any operation) are no longer processed. The follower doesn't recover from this.

      I've tested this out on 3.3 branch and confirmed this problem, however I haven't tried it on 3.4/3.5. It may not happen on the newer branches due to ZOOKEEPER-335, however there is certainly an issue with updating the "acceptedEpoch" files contained in the datadir. (I'll enter a separate jira for that)

      Attachments

        1. ZOOKEEPER-1277_br33.patch
          22 kB
          Patrick D. Hunt
        2. ZOOKEEPER-1277_br33.patch
          22 kB
          Patrick D. Hunt
        3. ZOOKEEPER-1277_br33.patch
          20 kB
          Patrick D. Hunt
        4. ZOOKEEPER-1277_br33.patch
          12 kB
          Patrick D. Hunt
        5. ZOOKEEPER-1277_br34.patch
          26 kB
          Patrick D. Hunt
        6. ZOOKEEPER-1277_br34.patch
          26 kB
          Patrick D. Hunt
        7. ZOOKEEPER-1277_trunk.patch
          25 kB
          Patrick D. Hunt
        8. ZOOKEEPER-1277_trunk.patch
          25 kB
          Patrick D. Hunt

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            phunt Patrick D. Hunt
            phunt Patrick D. Hunt
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment