[HDDS-4775] Avoid OM split brain going from 1 node OM to 3 node ratis without bootstrap

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Ethan Rose
• Created: 02/Feb/21 01:41
• Updated: 18/Feb/22 00:09

Description

The expected flow to add more OMs to a single node OM ratis cluster is to adjust the configurations to add the new OMs, and then start the new OMs in bootstrap mode (supply the -–bootstrap flag on startup) so that they will get all transaction history from the original OM. However, it is possible for the user to mistakenly adjust the configs and not start the new OMs with the --bootstrap command. This will cause the two new OMs to form their own Ratis ring with a 2/3 majority that can service write requests, while the original OM is still the leader of a single node Ratis ring and also servicing write requests. This leads to a split brain scenario that is difficult to detect without inspecting the logs, because all OMs appear functional and the cluster is writeable.

This Jira aims to detect such a scenario and shut down the OMs when it occurs, instructing the user to bootstrap the new OMs on startup instead.