[IGNITE-20317] Meta storage invokes are not completed when events are handled in DZM

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0
• Labels: ignite-3
• Epic Link: Recovery of DZM
• Assignee: Mirza Aliev
• Reporter: Sergey Uttsel
• Reviewer: Sergey Uttsel
• Created: 31/Aug/23 06:51
• Updated: 04/Sep/24 16:10
• Resolved: 16/Oct/23 07:11

Description

Motivation

There are meta storage invokes in DistributionZoneManager in zone's lifecycle. The futures of these invokes are ignored, so after the lifecycle method is completed actually not all its actions are completed. Therefore several invokes for example on createZone and alterZone can be reordered. Currently it does the meta storage invokes in:

1. ZonesConfigurationListener#onCreate to init a zone.
2. ZonesConfigurationListener#onDelete to clean up the zone data.
3. DistributionZoneManager#onUpdateFilter to save data nodes in the meta storage.
4. DistributionZoneManager#onUpdateScaleUp
5. DistributionZoneManager#onUpdateScaleDown
   DistributionZoneRebalanceEngine#onUpdateReplicas to apdate assignment on replicas update.
   LogicalTopologyEventListener to update logical topology.
   DistributionZoneRebalanceEngine#createDistributionZonesDataNodesListener watch listener to update pending assignments.

Definition of Done

Need to ensure event handling linearization. All immediate data nodes recalculation must be returned to the event handler.

Implementation Notes

• ZonesConfigurationListener#onCreate, ZonesConfigurationListener#onDelete, DistributionZoneManager#onUpdateFilter and DistributionZoneRebalanceEngine#onUpdateReplicas are invoked in configuration listeners. So we can just return the ms invoke future from these methods and it ensure, that this invoke will be completed within the current event handling.

• We cannnot return future from LogicalTopologyEventListener's methods. We can ignore these futures. It has drawback: we can skip the topology update

1. topology=[A,B], dataNodes=[A,B], scaleUp=0, scaleDown=100
2. Node C was joined to the topology and left quickly and ms invokes to update topology entry was reordered.
3. data nodes was not updated immediately to [A,B,C].
   We think that we can ignore this bug because eventually it doesn't break the consistency of the date node. For this purpose we need to change the invoke condition:
   `value(zonesLogicalTopologyVersionKey()).lt(longToBytes(newTopology.version()))` instead of
   `value(zonesLogicalTopologyVersionKey()).eq(longToBytes(newTopology.version() - 1))`

• Need to return ms invoke futures from WatchListener#onUpdate method of the data nodes listener.

Issue Links

• is a child of: IGNITE-17924 Core distributions zones functionality.
• relates to: IGNITE-20310 Meta storage invokes are not completed when DZM start is completed
• links to: GitHub Pull Request #2685