[IGNITE-18763] Parallel Meta Storage Watch processing

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0
• Component/s: persistence
• Labels: ignite-3
• Assignee: Aleksandr Polovtsev
• Reporter: Sergey Chugunov
• Created: 08/Feb/23 13:52
• Updated: 04/Sep/24 16:10
• Resolved: 14/Mar/23 17:44

Description

Currently Meta Storage Watches follow the given contract:

1. All listeners are processed sequentially in the order that they have been registered;
2. All listeners are processed in the same thread;
3. All notifications are grouped by a particular revision. After all events for a particular revision get processed, all modified keys are persisted into the Vault.

Some components may rely on this to ensure that dependent components can observe the most recent component state, because they registered their Meta Storage listeners after the component they depend on, therefore it is guaranteed that that component has already processed current Meta Storage revision.

This approach can be improved in terms of performance by enabling concurrent Watch processing in multiple threads. In order to do that, we need to check and fix the following places:

1. If a component depends on another component's state, this state must be extracted into a Versioned Value;
2. If a component updates its state, relying on the fact, that this state is always updated in a single thread, concurrent primitives must be introduced to ensure thread safety;
3. Check if it is possible to persist updated keys in the Vault individually, thus making it possible not to wait for all Watches for a particular revision to complete before starting to process updates belonging to the next revision.

Issue Links

• causes: IGNITE-19129 Remove parallel Watch processing
• links to: GitHub Pull Request #1748