[IGNITE-16011] Start new rebalance round, when partition assignments updated

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha5
• Labels: ignite-3
• Epic Link: Data rebalancing
• Assignee: Kirill Gusakov
• Reporter: Kirill Gusakov
• Reviewer: Alexander Lapin
• Created: 28/Nov/21 19:30
• Updated: 07/Jun/22 12:58
• Resolved: 04/May/22 12:13

Description

When partition assignments are updated, we need to start raft changePeers and handle failover scenarios.

When metastore event about partition assignments updates received we need to:

• Start all needed nodes 

   partition.assignments.pending / partition.assignments.stable

• After successful starts - check if current node is the leader of raft group (leader response must be updated by current term) and changePeers(leaderTerm, peers). changePeers from old terms must be skipped.

Also, we need the propagation of some new events from the raft side:

• onLeaderElected(boolean configurationChangeInProgress) - must be executed from the new leader when raft group changes the leader. Maybe we actually need to also check if a new lease is received - we need to investigate.
• onChangePeersError(errorContext) - must be executed when any errors during changePeers occurred
• onChangePeersCommitted(peers) - must be executed with the list of new peers when changePeers has successfully done.

and handle them by appropriate way:

• onLeaderElected(configurationChangeInProgress) - we need to:
  • if configurationChangeInProgress == false and pending/planned assignments not empty - run new changePeers. If true, do nothing.}}
• onChangePeersError(errorContext) - run failover logic
• onChangePeersCommitted(peers) - check if planned assignments is not empty and move it to pending.

• • Update pending and stable partitions assignments:

    metastoreInvoke: \\ atomic
        // Here we can check invariant that pending is empty, but planned is not is impossible
        partition.assignments.stable = appliedPeers
        if empty(partition.assignments.planned):
            partition.assignments.pending = empty
        else:
            partition.assignments.pending = partition.assignments.planned 

When partition.assignments.stable updated, we need to:

• Replace current raft client with new one, with appropriate peers
• Stop unneeded raft node
   

(Phase 1)

Issue Links

• links to: GitHub Pull Request #759