[FLINK-25981] ZooKeeperMultipleComponentLeaderElectionDriverTest.testLeaderElectionWithMultipleDrivers failed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.15.0
• Component/s: Runtime / Coordination
• Labels: test-stability
• Assignee: Matthias Pohl
• Reporter: Matthias Pohl
• Created: 07/Feb/22 13:24
• Updated: 18/Feb/22 09:39
• Resolved: 18/Feb/22 09:39

Description

We experienced a build failure in ZooKeeperMultipleComponentLeaderElectionDriverTest.testLeaderElectionWithMultipleDrivers. The test halted when waiting for the next leader in ZooKeeperMultipleComponentLeaderElectionDriverTest:256

Feb 04 18:02:54 "main" #1 prio=5 os_prio=0 tid=0x00007fab0800b800 nid=0xe07 waiting on condition [0x00007fab12574000]
Feb 04 18:02:54    java.lang.Thread.State: WAITING (parking)
Feb 04 18:02:54 	at sun.misc.Unsafe.park(Native Method)
Feb 04 18:02:54 	- parking to wait for  <0x000000008065c5c8> (a java.util.concurrent.CompletableFuture$Signaller)
Feb 04 18:02:54 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
Feb 04 18:02:54 	at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
Feb 04 18:02:54 	at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
Feb 04 18:02:54 	at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
Feb 04 18:02:54 	at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
Feb 04 18:02:54 	at org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriverTest.testLeaderElectionWithMultipleDrivers(ZooKeeperMultipleComponentLeaderElectionDriverTest.java:256)
[...]

The extended Maven logs indicate that the timeout happened while waiting for the second leader to be selected.

Test org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriverTest.testLeaderElectionWithMultipleDrivers is running.
--------------------------------------------------------------------------------
17:15:10,437 [           Thread-16] INFO  org.apache.curator.test.TestingZooKeeperMain                 [] - Starting server
17:15:10,450 [                main] INFO  org.apache.flink.runtime.util.ZooKeeperUtils                 [] - Enforcing default ACL for ZK connections
17:15:10,451 [                main] INFO  org.apache.flink.runtime.util.ZooKeeperUtils                 [] - Using '/flink/default' as Zookeeper namespace.
17:15:10,452 [                main] INFO  org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl [] - Starting
17:15:10,455 [                main] INFO  org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl [] - Default schema
17:15:10,462 [    main-EventThread] INFO  org.apache.flink.shaded.curator5.org.apache.curator.framework.state.ConnectionStateManager [] - State change: CONNECTED
17:15:10,467 [    main-EventThread] INFO  org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker [] - New config event received: {}
17:15:10,482 [Curator-ConnectionStateManager-0] DEBUG org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriver [] - Connected to ZooKeeper quorum. Leader election can start.
17:15:10,483 [Curator-ConnectionStateManager-0] DEBUG org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriver [] - Connected to ZooKeeper quorum. Leader election can start.
17:15:10,483 [Curator-ConnectionStateManager-0] DEBUG org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriver [] - Connected to ZooKeeper quorum. Leader election can start.
17:15:10,484 [    main-EventThread] INFO  org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker [] - New config event received: {}
17:15:10,562 [    main-EventThread] DEBUG org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriver [] - ZooKeeperMultipleComponentLeaderElectionDriver obtained the leadership.
17:15:10,600 [                main] INFO  org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriver [] - Closing ZooKeeperMultipleComponentLeaderElectionDriver.

Issue Links

• Discovered while testing: FLINK-25955 Add JobManagerRunner implementation that picks up dirty job results to be cleaned up
• is caused by: FLINK-24038 DispatcherResourceManagerComponent fails to deregister application if no leading ResourceManager
• is related to: FLINK-25847 KubernetesHighAvailabilityRecoverFromSavepointITCase. testRecoverFromSavepoint failed on azure
• is related to: FLINK-26223 Making ZK-related logs available in tests