[FLINK-24357] ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled fails with an Unhandled error - ASF JIRA

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 1.15.0
Fix Version/s: 1.14.3, 1.15.0
Component/s: Runtime / Coordination
Labels:
- pull-request-available
- test-stability

Description

In a private azure build when testing my own PR I've noticed the following error that looks unrelated to any of my changes (modifications to Task class error/cancellation handling logic):

2021-09-22T08:09:16.6244936Z Sep 22 08:09:16 [ERROR] testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled  Time elapsed: 28.753 s  <<< FAILURE!
2021-09-22T08:09:16.6245821Z Sep 22 08:09:16 java.lang.AssertionError: The TestingFatalErrorHandler caught an exception.
2021-09-22T08:09:16.6246513Z Sep 22 08:09:16 	at org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:78)
2021-09-22T08:09:16.6247281Z Sep 22 08:09:16 	at org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:33)
2021-09-22T08:09:16.6248167Z Sep 22 08:09:16 	at org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:57)
2021-09-22T08:09:16.6248862Z Sep 22 08:09:16 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2021-09-22T08:09:16.6249620Z Sep 22 08:09:16 	at org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2021-09-22T08:09:16.6250210Z Sep 22 08:09:16 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
2021-09-22T08:09:16.6250773Z Sep 22 08:09:16 	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2021-09-22T08:09:16.6251375Z Sep 22 08:09:16 	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
2021-09-22T08:09:16.6251951Z Sep 22 08:09:16 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
2021-09-22T08:09:16.6252562Z Sep 22 08:09:16 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
2021-09-22T08:09:16.6253415Z Sep 22 08:09:16 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
2021-09-22T08:09:16.6254469Z Sep 22 08:09:16 	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
2021-09-22T08:09:16.6255039Z Sep 22 08:09:16 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
2021-09-22T08:09:16.6256238Z Sep 22 08:09:16 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
2021-09-22T08:09:16.6257109Z Sep 22 08:09:16 	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
2021-09-22T08:09:16.6257766Z Sep 22 08:09:16 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
2021-09-22T08:09:16.6258406Z Sep 22 08:09:16 	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2021-09-22T08:09:16.6259050Z Sep 22 08:09:16 	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
2021-09-22T08:09:16.6259827Z Sep 22 08:09:16 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
2021-09-22T08:09:16.6260963Z Sep 22 08:09:16 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
2021-09-22T08:09:16.6261796Z Sep 22 08:09:16 	at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
2021-09-22T08:09:16.6262428Z Sep 22 08:09:16 	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
2021-09-22T08:09:16.6263268Z Sep 22 08:09:16 	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
2021-09-22T08:09:16.6263875Z Sep 22 08:09:16 	at java.util.Iterator.forEachRemaining(Iterator.java:116)
2021-09-22T08:09:16.6265025Z Sep 22 08:09:16 	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
2021-09-22T08:09:16.6265940Z Sep 22 08:09:16 	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
2021-09-22T08:09:16.6266767Z Sep 22 08:09:16 	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
2021-09-22T08:09:16.6267470Z Sep 22 08:09:16 	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
2021-09-22T08:09:16.6268165Z Sep 22 08:09:16 	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
2021-09-22T08:09:16.6269341Z Sep 22 08:09:16 	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2021-09-22T08:09:16.6269928Z Sep 22 08:09:16 	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
2021-09-22T08:09:16.6270951Z Sep 22 08:09:16 	at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
2021-09-22T08:09:16.6271683Z Sep 22 08:09:16 	at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
2021-09-22T08:09:16.6274483Z Sep 22 08:09:16 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
2021-09-22T08:09:16.6275060Z Sep 22 08:09:16 	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
2021-09-22T08:09:16.6275819Z Sep 22 08:09:16 	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
2021-09-22T08:09:16.6276557Z Sep 22 08:09:16 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
2021-09-22T08:09:16.6277234Z Sep 22 08:09:16 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
2021-09-22T08:09:16.6277770Z Sep 22 08:09:16 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
2021-09-22T08:09:16.6278345Z Sep 22 08:09:16 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
2021-09-22T08:09:16.6278909Z Sep 22 08:09:16 	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2021-09-22T08:09:16.6279433Z Sep 22 08:09:16 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2021-09-22T08:09:16.6279939Z Sep 22 08:09:16 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2021-09-22T08:09:16.6280627Z Sep 22 08:09:16 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2021-09-22T08:09:16.6281222Z Sep 22 08:09:16 Caused by: org.apache.flink.runtime.leaderelection.LeaderElectionException: Unhandled error in ZooKeeperLeaderElectionDriver: Ensure path threw exception
2021-09-22T08:09:16.6281884Z Sep 22 08:09:16 	at org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionDriver.unhandledError(ZooKeeperLeaderElectionDriver.java:295)
2021-09-22T08:09:16.6282527Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:713)
2021-09-22T08:09:16.6283172Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:709)
2021-09-22T08:09:16.6283803Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100)
2021-09-22T08:09:16.6284454Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
2021-09-22T08:09:16.6285302Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92)
2021-09-22T08:09:16.6286162Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.logError(CuratorFrameworkImpl.java:708)
2021-09-22T08:09:16.6286836Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:100)
2021-09-22T08:09:16.6287505Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:731)
2021-09-22T08:09:16.6288396Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.WatcherRemovalFacade.fixForNamespace(WatcherRemovalFacade.java:170)
2021-09-22T08:09:16.6289236Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:295)
2021-09-22T08:09:16.6289862Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:35)
2021-09-22T08:09:16.6290487Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.doRefreshData(TreeCache.java:287)
2021-09-22T08:09:16.6291319Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.refreshData(TreeCache.java:266)
2021-09-22T08:09:16.6291975Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.refresh(TreeCache.java:250)
2021-09-22T08:09:16.6292615Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.wasCreated(TreeCache.java:316)
2021-09-22T08:09:16.6293277Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCache.handleStateChange(TreeCache.java:819)
2021-09-22T08:09:16.6293918Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCache.access$1800(TreeCache.java:75)
2021-09-22T08:09:16.6294555Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCache$1.stateChanged(TreeCache.java:543)
2021-09-22T08:09:16.6295373Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$1(ConnectionStateManager.java:280)
2021-09-22T08:09:16.6296073Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:93)
2021-09-22T08:09:16.6296829Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:90)
2021-09-22T08:09:16.6297559Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:89)
2021-09-22T08:09:16.6298402Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:280)
2021-09-22T08:09:16.6299099Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:44)
2021-09-22T08:09:16.6300215Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:133)
2021-09-22T08:09:16.6318237Z Sep 22 08:09:16 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2021-09-22T08:09:16.6318982Z Sep 22 08:09:16 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-09-22T08:09:16.6323311Z Sep 22 08:09:16 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-09-22T08:09:16.6326943Z Sep 22 08:09:16 	at java.lang.Thread.run(Thread.java:748)
2021-09-22T08:09:16.6327544Z Sep 22 08:09:16 Caused by: org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /flink
2021-09-22T08:09:16.6328187Z Sep 22 08:09:16 	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
2021-09-22T08:09:16.6328785Z Sep 22 08:09:16 	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
2021-09-22T08:09:16.6329526Z Sep 22 08:09:16 	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
2021-09-22T08:09:16.6330053Z Sep 22 08:09:16 	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1139)
2021-09-22T08:09:16.6330609Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291)
2021-09-22T08:09:16.6331177Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
2021-09-22T08:09:16.6331842Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64)
2021-09-22T08:09:16.6332463Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
2021-09-22T08:09:16.6333051Z Sep 22 08:09:16 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
2021-09-22T08:09:16.6333499Z Sep 22 08:09:16 	... 22 more

2021-09-22T08:29:57.2073586Z Sep 22 08:29:57 [ERROR] Failures: 
2021-09-22T08:29:57.2074275Z Sep 22 08:29:57 [ERROR]   ZooKeeperLeaderElectionConnectionHandlingTest.testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled The TestingFatalErrorHandler caught an exception.