[IGNITE-22833] Test DistributionZoneCausalityDataNodesTest.testEmptyDataNodesOnZoneCreationBeforeTopologyEventAndZoneInitialisation is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1
• Labels: ignite-3
• Epic Link: Data Consistency TC Green Again Backlog
• Assignee: Vladislav Pyatkov
• Reporter: Vladislav Pyatkov
• Reviewer: Alexander Lapin
• Created: 25/Jul/24 09:39
• Updated: 12/Sep/24 17:04
• Resolved: 12/Sep/24 17:04

Description

[10:10:14] :             [:ignite-distribution-zones:test] DistributionZoneCausalityDataNodesTest > testEmptyDataNodesOnZoneCreationBeforeTopologyEventAndZoneInitialisation(int, int) > [1] 1, 1 STANDARD_ERROR
[10:10:14] :             [:ignite-distribution-zones:test]     [2024-07-23T10:10:14,100][WARN ][%test%metastorage-watch-executor-2][UpdateLogImpl] Unable to process catalog event
[10:10:14] :             [:ignite-distribution-zones:test]     java.lang.NullPointerException: null
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.distributionzones.DistributionZoneManager.initDataNodesAndTriggerKeysInMetaStorage(DistributionZoneManager.java:527) ~[ignite-distribution-zones-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.distributionzones.DistributionZoneManager.onCreateZone(DistributionZoneManager.java:456) ~[ignite-distribution-zones-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.distributionzones.DistributionZoneManager.lambda$registerCatalogEventListenersOnStartManagerBusy$37(DistributionZoneManager.java:1396) ~[ignite-distribution-zones-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.util.IgniteUtils.inBusyLock(IgniteUtils.java:832) ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.distributionzones.DistributionZoneManager.lambda$registerCatalogEventListenersOnStartManagerBusy$38(DistributionZoneManager.java:1395) ~[ignite-distribution-zones-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.event.AbstractEventProducer.fireEvent(AbstractEventProducer.java:88) ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.catalog.CatalogManagerImpl.access$000(CatalogManagerImpl.java:81) ~[ignite-catalog-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.catalog.CatalogManagerImpl$OnUpdateHandlerImpl.handle(CatalogManagerImpl.java:577) ~[ignite-catalog-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.catalog.CatalogManagerImpl$OnUpdateHandlerImpl.handle(CatalogManagerImpl.java:544) ~[ignite-catalog-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.catalog.storage.UpdateLogImpl$UpdateListener.onUpdate(UpdateLogImpl.java:320) ~[ignite-catalog-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.metastorage.server.Watch.onUpdate(Watch.java:67) ~[ignite-metastorage-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyWatches(WatchProcessor.java:245) ~[ignite-metastorage-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$notifyWatches$4(WatchProcessor.java:193) ~[ignite-metastorage-3.0.0-SNAPSHOT.jar:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) [?:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
[10:10:14] :             [:ignite-distribution-zones:test]      at java.base/java.lang.Thread.run(Thread.java:834) [?:?]

UPD: After some investigation, I would suggest to rewrite this test, so we don't use mocks of metastorage and don't block some metastorage updates and wait for further metastorage intereactions, like catalog updates. As an idea, we could use org.apache.ignite.internal.test.WatchListenerInhibitor to block reaction on changing of org.apache.ignite.internal.distributionzones.DistributionZonesUtil#zonesLogicalTopologyKey so that means that we can skip data nodes update as we wanted in the test

Issue Links

• links to: GitHub Pull Request #4337