[IGNITE-14209] Data rebalancing

• Type: Epic
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha5
• Labels: ignite-3
• Epic Name: Data rebalancing
• Assignee: Kirill Gusakov
• Reporter: Vyacheslav Koptilin
• Created: 18/Feb/21 16:20
• Updated: 07/Feb/23 14:14
• Resolved: 06/Jun/22 22:57

Description

Overview
The lifecycle of the Ignite cluster assumes that new nodes can join this cluster and the old ones can leave it. In order for the data to remain distributed equally in the cluster, some replicas have to be moved from one Ignite node to another. This process is called data rebalancing.
The main goal of this umbrella ticket is an approach that can be implemented based on Ignite Raft module.

 

The first implementation phase has the following constraints in order to simplify the rebalance process until the table group concept is not defined:

• only "empty" nodes can join the cluster (in other words, node restart with already exists data is not supported)
• rebalance can be triggered by NODE_JOIN/NODE_FAIL events only in an automatic way (alternative option is to introduce IgniteCluster interface which allows triggering rebalance in a manual way)
• reconfiguration of meta-storage nodes is not supported

Phase 1 includes the following sub-tasks:

• IGNITE-15125
• IGNITE-15491
• IGNITE-15490
• IGNITE-15288

Issue Links

• depends upon: IGNITE-15260 New peer cannot be added to a raft group via RaftGroupService
• depends upon: IGNITE-14790 Replace a storage map in RAFT partition listener to the persistence storage
• depends upon: IGNITE-15288 Fully implement RaftGroupService
• depends upon: IGNITE-15402 Cleanup storage directories if they are no longer needed.
• links to: GitHub Pull Request #839