[IGNITE-19955] Fix create zone on restart rewrites existing data nodes because of trigger key inconsistnecy

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0
• Labels: ignite-3
• Epic Link: Recovery of DZM
• Assignee: Mirza Aliev
• Reporter: Mirza Aliev
• Reviewer: Sergey Uttsel
• Created: 11/Jul/23 12:34
• Updated: 04/Sep/24 16:10
• Resolved: 14/Dec/23 09:04

Description

Outdated, see UPD

Currently we have the logic for initialisation of newly created zone that it writes keys

zoneDataNodesKey(zoneId), zoneScaleUpChangeTriggerKey(zoneId), zoneScaleDownChangeTriggerKey(zoneId), zonesChangeTriggerKey(zoneId)

to metastorage, and condition is

    static CompoundCondition triggerKeyConditionForZonesChanges(long revision, int zoneId) {
        return or(
                notExists(zonesChangeTriggerKey(zoneId)),
                value(zonesChangeTriggerKey(zoneId)).lt(ByteUtils.longToBytes(revision))
        );

Recovery process implies that the create zone event will be processed again, but with the higher revision, so data nodes will be rewritten.

We need to handle this situation, so data nodes will be consistent after restart.

Possible solution is to change condition to

    static SimpleCondition triggerKeyConditionForZonesCreation(long revision, int zoneId) {
        return notExists(zonesChangeTriggerKey(zoneId));
    }

    static SimpleCondition triggerKeyConditionForZonesDelete(int zoneId) {
        return exists(zonesChangeTriggerKey(zoneId));
    }

so we could not rely on revision and check only existence of the key, when we create or remove zone. The problem in this solution is that reordering of the create and remove on some node could lead to not consistent state for zones key in metastorage

UPD:

This problem will be resolves once we implement https://issues.apache.org/jira/browse/IGNITE-20561
In this ticket we need to unmute all tickets that were muted by this ticket

Issue Links

• is blocked by: IGNITE-20561 Change condition for DistributionZonesUtil#triggerKeyConditionForZonesChanges to use ConditionType#TOMBSTONE
• is blocked by: IGNITE-19967 Design restart of Distribution Zone Manager in accordance with the new local recovery process
• relates to: IGNITE-19778 Restore components state on metastorage recovery
• links to: GitHub Pull Request #2927