[IGNITE-18079] Integrate RAFT streaming snapshots

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0
• Labels: ignite-3
• Epic Link: Storage support for Rebalancing
• Assignee: Roman Puchkovskiy
• Reporter: Roman Puchkovskiy
• Reviewer: Kirill Tkalenko
• Created: 03/Nov/22 14:33
• Updated: 04/Sep/24 16:20
• Resolved: 04/Jan/23 09:12

Description

Currently, the code implementing sending and receiving of snapshots is already there. Some scenarios need to be implemented:

1. Node that has data restarts successfully (trivial, but should make sure that 'startup snapshot' is 'applied' correctly)
2. Start a cluster of 3 nodes, stop one of them, add some data, provoke the lagging node to restore itself from a snapshot, start it, make sure it catches up
3. Test that a node fed with a snapshot can act as a leader later (including installing snapshots to other nodes)
4. Test that a node that catches up using a snapshot gets updates happening concurrently with snapshot streaming, applied in the correct order

Issue Links

• causes: IGNITE-18122 Track last applied term and group config in storages
• is blocked by: IGNITE-18421 Fix IncomingSnapshotCopier#cancel for the case of network problems
• is blocked by: IGNITE-18495 Fix RAFT snapshot installation hang due to response swap on retry
• is related to: IGNITE-18170 Deadlock in TableManager#updateAssignmentInternal()
• is related to: IGNITE-18428 After a RAFT snapshot install timed out, subsequent installs consistently failed
• is related to: IGNITE-18432 SafeTimeCandidateManager#commitIndex() fails assertion if indices have different terms
• is related to: IGNITE-18449 Select from a table fails after the table is created
• is related to: IGNITE-18451 Node stop may hang if an index is not created yet
• is related to: IGNITE-19233 Add integration test for adding new node to the cluster while installing RAFT snapshot
• is related to: IGNITE-18430 Add integration test for a case when RAFT snapshot installation is cancelled in the middle
• relates to: IGNITE-18485 ItTableRaftSnapshotsTest. entriesKeepAppendedAfterSnapshotInstallation() sometimes hangs
• split from: IGNITE-17262 Implement RAFT snapshot streamer
• links to: GitHub Pull Request #1461