[HDDS-7706] OM Bootstrap is unable to add a node to a raft ring

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: OM HA
• Target Version/s: 1.4.0
• Assignee: Kirill Sizov
• Reporter: Kirill Sizov
• Created: 23/Dec/22 11:47
• Updated: 16/Jan/23 11:36
• Resolved: 16/Jan/23 11:36

Description

During the lifetime of OM HA the nodes might receive peer list update messages. It usually happens when a node goes down and the up. But as long as all the nodes of the cluster are present in this list - everything's fine.

However if we bootstrap a new node, in the following example it is om4, and it replays the raft log on its side, such message will be fatal and cause it to exit.

2022-12-22 19:45:17,386 [Thread[Thread-21,5,main]] INFO security.OzoneDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2022-12-22 19:45:17,386 [Listener at om4address/9862] INFO om.OzoneManager: Version File has different layout version (3) than OM DB (null). That is expected if this OM has never been finalized to a newer layout version.
2022-12-22 19:45:17,387 [om4@group-942F8267F22A-StateMachineUpdater] INFO ratis.OzoneManagerStateMachine: Received Configuration change notification from Ratis. New Peer list:
[id: "om1"
address: "om1address:9872"
, id: "om3"
address: "om3address:9872"
, id: "om2"
address: "om2address:9872"
]
2022-12-22 19:45:17,387 [om4@group-942F8267F22A-StateMachineUpdater] ERROR om.OzoneManager: Fatal Error: Shutting down as OM has been decommissioned.
2022-12-22 19:45:17,388 [om4@group-942F8267F22A-StateMachineUpdater] ERROR om.OzoneManager: Terminating with exit status 1: Shutting down as OM has been decommissioned.

Expected behavior:
the new node should be able to correctly replay the raft log.

Issue Links

• duplicates: HDDS-7584 Addition of new OM node expels itself from the Ratis ring after restart
• relates to: HDDS-4330 Bootstrap new OM node