[FLINK-31169] KubernetesResourceManagerDriverTest.testOnPodDeleted fails fatally due to 239 exit code

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.17.0
• Fix Version/s: 1.17.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available test-stability
• Assignee: Xintong Song
• Reporter: Matthias Pohl
• Created: 21/Feb/23 12:57
• Updated: 22/Feb/23 10:06
• Resolved: 22/Feb/23 09:56

Description

master: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46341&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=5acec1b4-945b-59ca-34f8-168928ce5199&l=27329

[...]
Feb 21 04:44:11 [ERROR] Process Exit Code: 239
Feb 21 04:44:11 [ERROR] Crashed tests:
Feb 21 04:44:11 [ERROR] org.apache.flink.kubernetes.KubernetesResourceManagerDriverTest
Feb 21 04:44:11 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748)
[...]

[...]
Test org.apache.flink.kubernetes.KubernetesResourceManagerDriverTest.testOnPodDeleted[testOnPodDeleted()] is running.
--------------------------------------------------------------------------------
04:43:57,681 [ForkJoinPool-4-worker-1] INFO  org.apache.flink.kubernetes.KubernetesResourceManagerDriver  [] - Recovered 0 pods from previous attempts, current attempt id is 1.
04:43:57,701 [testing-rpc-main-thread] INFO  org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: []
04:43:57,705 [testing-rpc-main-thread] INFO  org.apache.flink.kubernetes.KubernetesResourceManagerDriver  [] - Creating new TaskManager pod with name testing-flink-cluster-taskmanager-1-1 and resource <704,0.0>.
04:43:57,708 [testing-rpc-main-thread] INFO  org.apache.flink.kubernetes.KubernetesResourceManagerDriver  [] - Received new TaskManager pod: testing-flink-cluster-taskmanager-1-1
04:43:57,708 [testing-rpc-main-thread] INFO  org.apache.flink.kubernetes.KubernetesResourceManagerDriver  [] - Pod testing-flink-cluster-taskmanager-1-1 is created.
04:43:57,708 [testing-rpc-main-thread] WARN  org.apache.flink.kubernetes.KubernetesResourceManagerDriver  [] - Pod testing-flink-cluster-taskmanager-1-1 is terminated before being scheduled.
04:43:57,709 [testing-rpc-main-thread] ERROR org.apache.flink.kubernetes.KubernetesResourceManagerDriver  [] - Error completing resource request.
org.apache.flink.util.FlinkException: Pod is terminated.
        at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.onPodTerminated(KubernetesResourceManagerDriver.java:379) ~[classes/:?]
        at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.lambda$handlePodEventsInMainThread$2(KubernetesResourceManagerDriver.java:347) ~[classes/:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
04:43:57,724 [testing-rpc-main-thread] ERROR org.apache.flink.util.FatalExitExceptionHandler              [] - FATAL: Thread 'testing-rpc-main-thread' produced an uncaught exception. Stopping the process...
java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.flink.util.FlinkException: Pod is terminated.
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_292]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_292]
        at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:838) ~[?:1.8.0_292]
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[?:1.8.0_292]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_292]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_292]
        at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.onPodTerminated(KubernetesResourceManagerDriver.java:379) ~[classes/:?]
        at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.lambda$handlePodEventsInMainThread$2(KubernetesResourceManagerDriver.java:347) ~[classes/:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: java.lang.RuntimeException: org.apache.flink.util.FlinkException: Pod is terminated.
        at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) ~[flink-core-1.18-SNAPSHOT.jar:1.18-SNAPSHOT]
        at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.lambda$requestResource$1(KubernetesResourceManagerDriver.java:233) ~[classes/:?]
        at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) ~[?:1.8.0_292]
        ... 12 more
Caused by: org.apache.flink.util.FlinkException: Pod is terminated.
        ... 9 more
[...]

Issue Links

• is caused by: FLINK-30908 Fatal error in ResourceManager caused YARNSessionFIFOSecuredITCase.testDetachedMode to fail
• links to: GitHub Pull Request #21986