[IGNITE-18495] Fix RAFT snapshot installation hang due to response swap on retry

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0
• Labels: ignite-3
• Epic Link: Storage support for Rebalancing
• Assignee: Roman Puchkovskiy
• Reporter: Roman Puchkovskiy
• Reviewer: Mirza Aliev
• Created: 03/Jan/23 14:37
• Updated: 04/Sep/24 16:19
• Resolved: 04/Jan/23 07:40

Description

The scenario follows:

1. InstallSnapshot request is sent, its processing starts hanging forever (it will be cancelled on step 3)
2. After a timeout, second InstallSnapshot request is sent with same index+term as the first had; in JRaft, it causes a special handling (previous request processing is NOT cancelled)
3. After a timeout, third InstallSnapshot request is sent with DIFFERENT index, so it cancels the first snapshot processing effectively unblocking the first thread

In the original JRaft implementation, after being unblocked, the first thread fails to clean up, so subsequent retries will always see a phantom of an unfinished snapshot, so the snapshotting process will be jammed. Also, node stop might stuck because one 'download' task will remain unfinished forever.

Issue Links

• blocks: IGNITE-18079 Integrate RAFT streaming snapshots
• is duplicated by: IGNITE-18428 After a RAFT snapshot install timed out, subsequent installs consistently failed
• links to: GitHub Pull Request #1491