[CURATOR-622] Add Randomness to LeaderLatch Elections

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Recipes
• Assignee: Unassigned
• Reporter: Tim Black
• Created: 13/Nov/21 18:29
• Updated: 14/Nov/21 22:03

Description

Currently, LeaderLatch uses EPHEMERAL_SEQUENTIAL nodes, with the next leader chosen by the lowest numbered node. In a multi-server environment where each server is a participant in multiple elections, the result is that the leader will always be the server that has been up the longest.(Or first to be restarted during a rolling restart)

Instead of using sequentially numbered nodes, I propose instead that the node number for a new participant be created by adding a random number(From a constrained range) to the current leader number.(Defaults to zero) If a node with that number exists, repeat until an available node is found. After initial node creation, all other aspects of the leader election will remain unchanged.

I have an implementation for this that I am testing locally and will submit a PR once the tests are complete.