[HDDS-4136] Design for Error/Exception handling in state update for container/pipeline V2

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: SCM
• Assignee: Glen Geng
• Reporter: Glen Geng
• Created: 24/Aug/20 09:18
• Updated: 13/Feb/24 07:59

Description

I have a concern about how to handle exceptions occurred in writing RocksDB for container V2, such as allocateContainer, deleteContainer and updateContainerState.

For non-HA case, allocateContainer reverts the memory state changes if meet IOException for db operations. deleteContainer and updateContainerState just throw out the IOException and leave the memory state in an inconsistency state.

After we enable SCM-HA, if leader SCM succeed the operation, meanwhile any follower SCM fails due to db exception, what can we do to ensure that states of leader and followers won't diverge, a.k.a. ensure the replicated StateMachine for leader and followers ?

We have to ensure Atomicity of ACID for state update: If any exception occurred, SCM (no matter leader or follower) should throw exception and keep states unchanged. No partial change is allowed so that leader SCM can safely revert the state change for the whole raft groups.

Above analysis also applies to pipeline V2 and other issues besides disk failure.