[FLINK-22577] KubernetesLeaderElectionAndRetrievalITCase is failing

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.13.0, 1.14.0, 1.12.3
• Fix Version/s: 1.14.0, 1.13.1, 1.12.4
• Component/s: Deployment / Kubernetes, Runtime / Coordination
• Labels: pull-request-available test-stability
• Assignee: Till Rohrmann
• Reporter: Matthias Pohl
• Created: 05/May/21 16:00
• Updated: 28/Aug/21 11:19
• Resolved: 10/May/21 10:12

Description

KubernetesLeaderElectionAndRetrievalITCase is failing constantly. Running it locally results in an AssertionError:

3069 [KubernetesLeaderElector-ExecutorService-thread-1] DEBUG io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Loop thread interrupted
java.lang.InterruptedException: null
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_265]
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_265]
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) ~[?:1.8.0_265]
	at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200) ~[kubernetes-client-4.9.2.jar:?]
	at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100) ~[kubernetes-client-4.9.2.jar:?]
	at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71) ~[kubernetes-client-4.9.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
3078 [KubernetesLeaderElector-ExecutorService-thread-1] ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL: Thread 'KubernetesLeaderElector-ExecutorService-thread-1' produced an uncaught exception. Stopping the process...
java.lang.AssertionError: null
	at org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver.writeLeaderInformation(KubernetesLeaderElectionDriver.java:130) ~[classes/:?]
	at org.apache.flink.runtime.leaderelection.TestingLeaderElectionEventHandler.lambda$onRevokeLeadership$1(TestingLeaderElectionEventHandler.java:69) ~[test-classes/:?]
	at org.apache.flink.runtime.leaderelection.TestingLeaderElectionEventHandler.waitForInitialization(TestingLeaderElectionEventHandler.java:93) ~[test-classes/:?]
	at org.apache.flink.runtime.leaderelection.TestingLeaderElectionEventHandler.onRevokeLeadership(TestingLeaderElectionEventHandler.java:66) ~[test-classes/:?]
	at org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver$LeaderCallbackHandlerImpl.notLeader(KubernetesLeaderElectionDriver.java:202) ~[classes/:?]
	at io.fabric8.kubernetes.client.extended.leaderelection.LeaderCallbacks.onStopLeading(LeaderCallbacks.java:38) ~[kubernetes-client-4.9.2.jar:?]
	at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:72) ~[kubernetes-client-4.9.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_265]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_265]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]

The failure never popped up due to FLINK-22564

• 1.12 release branch
• 1.13 release branch
• master

Issue Links

• is related to: FLINK-22564 Kubernetes-related ITCases do not fail even in case of failure
• links to: Github PR #15849
• links to: GitHub Pull Request #15849