[ZOOKEEPER-917] Leader election selected incorrect leader

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 3.2.2
• Component/s: leaderElection, server

Environment

Cloudera distribution of zookeeper (patched to never cache DNS entries)
Debian lenny

• Assignee: Unassigned
• Reporter: Alexandre Hardy
• Created: 03/Nov/10 12:33
• Updated: 19/Nov/11 01:01
• Resolved: 04/Nov/10 12:40

Description

We had three nodes running zookeeper:

• 192.168.130.10
• 192.168.130.11
• 192.168.130.14

192.168.130.11 failed, and was replaced by a new node 192.168.130.13 (automated startup). The new node had not participated in any zookeeper quorum previously. The node 192.148.130.11 was permanently removed from service and could not contribute to the quorum any further (powered off).

DNS entries were updated for the new node to allow all the zookeeper servers to find the new node.

The new node 192.168.130.13 was selected as the LEADER, despite the fact that it had not seen the latest zxid.

This particular problem has not been verified with later versions of zookeeper, and no attempt has been made to reproduce this problem as yet.