[FLINK-10443] Flink Yarn CLI hanging after unsuccessful deployment

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Abandoned
• Affects Version/s: 1.5.4, 1.9.1
• Component/s: Command Line Client, Deployment / YARN
• Labels: auto-deprioritized-major
• Assignee: Unassigned
• Reporter: Nico Kruber
• Created: 26/Sep/18 15:25
• Updated: 11/Aug/21 12:32
• Resolved: 11/Aug/21 12:32

Description

It looks like the Flink CLI is retrying forever to cancel a YARN deployment which has failed already (during deployment). Additionally, you cannot cancel the CLI via CTRL+C

Reproduce via a failing deployment due to an inaccessible SSL key file:

// flink-conf.yaml
security.ssl.enabled: true
web.ssl.enabled: false
security.ssl.keystore: /path/to/non-existing/internal.keystore
security.ssl.keystore-password: internal_store_password
security.ssl.key-password: internal_key_password
security.ssl.truststore: /path/to/non-existing/internal.keystore
security.ssl.truststore-password: internal_store_password

./bin/flink run -m yarn-cluster -p 2 ./examples/streaming/WordCount.jar --input /usr/share/doc/java-1.8.0-openjdk-devel-1.8.0.181/LICENSE

After the failure, the CLI will hang with repeatedly printing this:

2018-09-26 15:17:50,348 INFO  org.apache.hadoop.io.retry.RetryInvocationHandler             - Exception while invoking ApplicationClientProtocolPBClientImpl.forceKillApplication over null. Retrying after sleeping for 30000ms.
java.io.IOException: The client is stopped
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1519)
        at org.apache.hadoop.ipc.Client.call(Client.java:1381)
        at org.apache.hadoop.ipc.Client.call(Client.java:1345)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
        at com.sun.proxy.$Proxy8.forceKillApplication(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.forceKillApplication(ApplicationClientProtocolPBClientImpl.java:213)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
        at com.sun.proxy.$Proxy9.forceKillApplication(Unknown Source)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.killApplication(YarnClientImpl.java:439)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.killApplication(YarnClientImpl.java:419)
        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.failSessionDuringDeployment(AbstractYarnClusterDescriptor.java:1236)
        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.access$200(AbstractYarnClusterDescriptor.java:111)
        at org.apache.flink.yarn.AbstractYarnClusterDescriptor$DeploymentFailureHook.run(AbstractYarnClusterDescriptor.java:1493)