[FLINK-30908] Fatal error in ResourceManager caused YARNSessionFIFOSecuredITCase.testDetachedMode to fail

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.17.0
• Fix Version/s: 1.17.0
• Component/s: Deployment / YARN, Runtime / Coordination
• Labels: pull-request-available test-stability
• Assignee: Xintong Song
• Reporter: Matthias Pohl
• Created: 06/Feb/23 07:28
• Updated: 03/May/23 15:03
• Resolved: 20/Feb/23 02:12

Description

There's a build failure in YARNSessionFIFOSecuredITCase.testDetachedMode which is caused by a fatal error in the ResourceManager:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45720&view=logs&j=245e1f2e-ba5b-5570-d689-25ae21e5302f&t=d04c9862-880c-52f5-574b-a7a79fef8e0f&l=29869

Feb 05 02:41:58 java.io.InterruptedIOException: Interrupted waiting to send RPC request to server
Feb 05 02:41:58 java.io.InterruptedIOException: Interrupted waiting to send RPC request to server
Feb 05 02:41:58 	at org.apache.hadoop.ipc.Client.call(Client.java:1480) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	at org.apache.hadoop.ipc.Client.call(Client.java:1422) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	at com.sun.proxy.$Proxy31.allocate(Unknown Source) ~[?:?]
Feb 05 02:41:58 	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) ~[hadoop-yarn-common-3.2.3.jar:?]
Feb 05 02:41:58 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
Feb 05 02:41:58 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_292]
Feb 05 02:41:58 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
Feb 05 02:41:58 	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
Feb 05 02:41:58 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	at com.sun.proxy.$Proxy32.allocate(Unknown Source) ~[?:?]
Feb 05 02:41:58 	at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:325) ~[hadoop-yarn-client-3.2.3.jar:?]
Feb 05 02:41:58 	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:311) [hadoop-yarn-client-3.2.3.jar:?]
Feb 05 02:41:58 Caused by: java.lang.InterruptedException
Feb 05 02:41:58 	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_292]
Feb 05 02:41:58 	at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_292]
Feb 05 02:41:58 	at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1180) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	at org.apache.hadoop.ipc.Client.call(Client.java:1475) ~[hadoop-common-3.2.3.jar:?]
Feb 05 02:41:58 	... 17 more

Issue Links

• causes: FLINK-31169 KubernetesResourceManagerDriverTest.testOnPodDeleted fails fatally due to 239 exit code
• causes: FLINK-31974 JobManager crashes after KubernetesClientException exception with FatalExitExceptionHandler
• is related to: FLINK-22312 YARNSessionFIFOSecuredITCase>YARNSessionFIFOITCase.checkForProhibitedLogContents due to the heartbeat exception with Yarn RM
• relates to: FLINK-31609 Fatal error in ResourceManager caused YARNSessionFIFOSecuredITCase.testDetachedMode to fail
• links to: GitHub Pull Request #21868