[IGNITE-18170] Deadlock in TableManager#updateAssignmentInternal()

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0
• Labels: ignite-3
• Epic Link: Storage support for Rebalancing
• Assignee: Roman Puchkovskiy
• Reporter: Roman Puchkovskiy
• Created: 16/Nov/22 07:52
• Updated: 04/Sep/24 16:19
• Resolved: 06/Apr/23 11:20

Description

Currently, TableManager#updateAssignmentsInternal is fully synchronous. The scenario is as follows:

1. updateAssignmentsInternal starts a RAFT group for a partition
2. FSMCallerImpl finds out that its applied index is below the group committed index, so it starts to apply the missing log entries in its init() method (this is still done synchronously)
3. While doing so, it invokes PartitionListener, which tries to execute an insert
4. To make an insert, a PK is needed, so it the insertion code tries to obtain a PK from its future like this: pkFuture.join()
5. That future is completed from IndexManager#createIndexLocally(), which is invoked by ConfigurationNotifier later than updateassignmentsInternal in the same thread
6. As a result, the PK future cannot be completed before the sync updateAssignmentsInternal finishes its job and returns, and it cannot finish its job before the PK future is completed

We should make updateAssignmentsInternal async. 

Issue Links

• is blocked by: IGNITE-18181 Race during table creation
• is blocked by: IGNITE-18203 Table/index dependecy deadlock on node restart
• is fixed by: IGNITE-18203 Table/index dependecy deadlock on node restart
• relates to: IGNITE-18079 Integrate RAFT streaming snapshots
• links to: GitHub Pull Request #1346