[ZOOKEEPER-1277] servers stop serving when lower 32bits of zxid roll over

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.3.3
• Fix Version/s: 3.3.5, 3.4.4, 3.5.0
• Component/s: server
• Hadoop Flags: Reviewed

Release Note

Workaround: there is a simple workaround for this issue. Force a leader re-election before the lower 32bits reach 0xffffffff

Most users won't even see this given the number of writes on a typical installation - say you are doing 500 writes/second, you'd see this after ~3 months if the quorum is stable, any changes (such as upgrading the server software) would cause the xid to be reset, thereby staving off this issue for another period.

• Assignee: Patrick D. Hunt
• Reporter: Patrick D. Hunt
• Created: 02/Nov/11 16:46
• Updated: 28/Oct/24 04:40
• Resolved: 15/Mar/12 16:55

Description

When the lower 32bits of a zxid "roll over" (zxid is a 64 bit number, however the upper 32 are considered the epoch number) the epoch number (upper 32 bits) are incremented and the lower 32 start at 0 again.

This should work fine, however in the current 3.3 branch the followers see this as a NEWLEADER message, which it's not, and effectively stop serving clients. Attached clients seem to eventually time out given that heartbeats (or any operation) are no longer processed. The follower doesn't recover from this.

I've tested this out on 3.3 branch and confirmed this problem, however I haven't tried it on 3.4/3.5. It may not happen on the newer branches due to ZOOKEEPER-335, however there is certainly an issue with updating the "acceptedEpoch" files contained in the datadir. (I'll enter a separate jira for that)

Issue Links

• Blocked: ZOOKEEPER-2789 Reassign `ZXID` for solving 32bit overflow problem
• is related to: ZOOKEEPER-4870 Proactive leadership transfer
• is related to: ZOOKEEPER-4883 Rollover leader epoch when counter part of zxid reach limit
• relates to: ZOOKEEPER-1278 acceptedEpoch not handling zxid rollover in lower 32bits
• relates to: ZOOKEEPER-3253 client should not send requests with cxid=-4, -2, or -1