[IGNITE-22246] Write a test making sure that double-voting problem is not possible for volatile partitions

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: ignite-3
• Assignee: Unassigned
• Reporter: Roman Puchkovskiy
• Created: 15/May/24 11:12
• Updated: 15/May/24 11:19

Description

IGNITE-16668 describes a design that is currently implemented in the PartitionReplicatorNodeRecovery class. This is needed to make sure that, when we restart an Ignite node having a volatile partition (which uses volatile Raft metastorage), and hence we break Raft assumptions about Raft metastorage being stored on a stable storage (i.e. its contents cannot be destroyed), we are still safe as the additional logic makes sure that the broken Raft assumptions do not cause any trouble.

The trouble that can be caused is the 'double-vote' problem. The scenario follows:

1. Node A starts election and proposes itself as a candidate
2. Node C votes for node A
3. Node C restart and loses votedFor in its Raft metastorage
4. Node B starts election and proposes itself as a candidate (in the same term in thich node A proposed itself)
5. Node C votes for B (as it forgot that it already voted for A in this term)
6. Both nodes A and B get elected as leaders, so 'at most one leader at a term' property of Raft is broken

Our test should try to get 2 leaders in the same term repeating this scenario. Probably, a lot of message cancellation/delaying will be needed (for example, to make sure that node B does not see node A self-proposals).

Issue Links

• is related to: IGNITE-16668 Design in-memory raft group reconfiguration on node failure