[CAMEL-5548] It is possible to get a split-brain scenario among applications managed by a zookeeper master election

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Later
• Affects Version/s: 2.10.0
• Fix Version/s: Future
• Component/s: camel-zookeeper
• Estimated Complexity: Moderate
• Assignee: Claus Ibsen
• Reporter: Andrew Wheat
• Created: 29/Aug/12 09:51
• Updated: 21/Sep/17 09:48
• Resolved: 21/Sep/17 09:48

Description

Setup:
Three Zookeeper instances (Z1, Z2, Z3) in two datacenters; A and B. Z1 and Z2 are in data center A; Z3 is in data center B.

Four Application servers (A1-A4) split evenly between the two datacenters, using a camel route to determine if the a message should be sent (quartz -> zookeeperpolicy -> activemq)

The master node is A3 in data center B.

When the network fails between the two data centers, The ZK servers in data center A have quorum.
When something causes an election to occur in data center A, one of A1 or A2 is elected.

Given that no election has occurred in data center B, A3 is also marked as master.

=========
Proposed solution:
A change in the zookeeper structure should trigger a re-election among the application servers. Since A3 and A4 will be connected to a zookeeper that does not have quorum, neither A3 or A4 will be elected as master.

Issue Links

• relates to: CAMEL-8208 ZooKeeperRoutePolicy is not able to recover after session expiration