[IGNITE-8020] Rebalancing for persistent caches should transfer file store over network instead of using existing supply/demand protocol

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: persistence
• Labels: iep-16 iep-28
• Assignee: Maxim Muzafarov
• Reporter: Ilya Lantukh
• Created: 22/Mar/18 13:17
• Updated: 05/Oct/23 14:55

Description

Existing rebalancing protocol is suitable for in-memory data storage, but for data persisted in files it is sub-optimal and requires a lot of unnecessary steps. Efforts to optimize it led to necessity to completely rework the protocol - instead of sending batches (SupplyMessages) with cache entries it is possible to send data files directly.

The algorithm should look like this:
1. Demander node sends requests with required partition IDs (like now)
2. Supplier node receives request and performs a checkpoint.
3. After checkpoint is done, supplier sends files with demanded partitions using low-level NIO API.
4. During steps 2-3, demander node should work in special mode - it should temporary store all incoming updates in such way that they can be quickly applied later.
5. After files are transferred, demander applies updates stored at step 4.

The tricky part here is to switch work modes of demander node avoiding all possible race conditions. Also, the aforementioned algorithm should be extended to transfer or rebuild query indexes.

Sub-Tasks

• [POC] Introduce mechanism to fetch partition file via a p2p protocol: Open
• Add support files transmission between nodes over connection via CommunicationSpi: Resolved
• Index rebuild procedure over cache partition file: Open
• Implement file rebalancing management: Open
• Provide benchmark scripts for the rebalance procedure test: Open
• Faster index eviction: In Progress
• Design an cluster traversal algorithm to preload cache groups as faster as possible: Open
• Make historical rebalance wokring per-partition level: Open
• Request snapshot from remote node: Patch Available