[IGNITE-18003] ItTablesApiTest#testCreateDropTable became flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-beta1
• Labels: ignite-3
• Assignee: Denis Chudov
• Reporter: Denis Chudov
• Reviewer: Alexander Lapin
• Created: 27/Oct/22 17:33
• Updated: 03/Nov/22 16:31
• Resolved: 03/Nov/22 16:30

Description

Motivation:
Seems that index create listener in IndexManager gets table from TableManager without using causality token, and sometimes it appears that table is not created yet. Using the causality token should gurantee the presence of table in table manager for the index manager.
This causes unstability of the test ItTablesApiTest#testCreateDropTable:

[16:41:04][org.apache.ignite:ignite-runner] 2022-10-27 19:41:04:009 +0300 [INFO][main][ItTablesApiTest] Dropping the table ignored.
[16:41:04][org.apache.ignite:ignite-runner] org.apache.ignite.lang.TableNotFoundException: IGN-TBL-2 TraceId:4ecb38fa-7440-448f-8d13-4d351df39ee7 The table does not exist [name="PUBLIC"."TBL1"]
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.table.distributed.TableManager.lambda$dropTableAsyncInternal$57(TableManager.java:1368)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.table.distributed.TableManager.dropTableAsyncInternal(TableManager.java:1364)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.table.distributed.TableManager.dropTableAsync(TableManager.java:1354)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.runner.app.ItTablesApiTest.dropTableIfExists(ItTablesApiTest.java:517)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.runner.app.ItTablesApiTest.testCreateDropTable(ItTablesApiTest.java:451)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
[16:41:04][org.apache.ignite:ignite-runner] 	Suppressed: java.lang.RuntimeException: This is a trimmed root
[16:41:04][org.apache.ignite:ignite-runner] 		at org.apache.ignite.internal.testframework.IgniteTestUtils.await(IgniteTestUtils.java:749)
[16:41:04][org.apache.ignite:ignite-runner] 		at org.apache.ignite.internal.testframework.IgniteTestUtils.await(IgniteTestUtils.java:769)
[16:41:04][org.apache.ignite:ignite-runner] 		... 72 more
[16:41:04][org.apache.ignite:ignite-runner] 2022-10-27 19:41:04:013 +0300 [INFO][vault1][ConfigurationRegistry] Failed to notify configuration listener
[16:41:04][org.apache.ignite:ignite-runner] java.util.concurrent.CompletionException: java.lang.NullPointerException
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:757)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.index.IndexManager.createIndexLocally(IndexManager.java:387)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.index.IndexManager.onIndexCreate(IndexManager.java:364)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.index.IndexManager$ConfigurationListener.onCreate(IndexManager.java:541)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier.notifyPublicListeners(ConfigurationNotifier.java:492)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier$1.visitNamedListNode(ConfigurationNotifier.java:205)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier$1.visitNamedListNode(ConfigurationNotifier.java:128)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.schema.configuration.TablesNode.traverseChildren(Unknown Source)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier.notifyListeners(ConfigurationNotifier.java:128)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier.notifyListeners(ConfigurationNotifier.java:90)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.ConfigurationRegistry$2.visitInnerNode(ConfigurationRegistry.java:310)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.ConfigurationRegistry$2.visitInnerNode(ConfigurationRegistry.java:292)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.SuperRoot.traverseChildren(SuperRoot.java:103)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.ConfigurationRegistry.notificator(ConfigurationRegistry.java:292)
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.configuration.ConfigurationChanger.lambda$updateFromListener$9(ConfigurationChanger.java:596)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1742)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.lang.Thread.run(Thread.java:834)
[16:41:04][org.apache.ignite:ignite-runner] Caused by: java.lang.NullPointerException
[16:41:04][org.apache.ignite:ignite-runner] 	at org.apache.ignite.internal.index.IndexManager.lambda$createIndexLocally$9(IndexManager.java:389)
[16:41:04][org.apache.ignite:ignite-runner] 	at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
[16:41:04][org.apache.ignite:ignite-runner] 	... 22 more

Implementation Notes
Getting table from TableManager using causality token guarantees that the future that will be returned to the listener in IndexManager will be completed when the corresponding table is created after the configuration changes related to this token are applied, and this future will contain this table as a result.

Issue Links

• links to: GitHub Pull Request #1266