[FLINK-9196] YARN: Flink binaries are not deleted from HDFS after cluster shutdown

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: Deployment / YARN
• Labels: flip-6
• Assignee: Gary Yao
• Reporter: Gary Yao
• Created: 17/Apr/18 15:06
• Updated: 02/May/18 10:37
• Resolved: 02/May/18 10:37

Description

When deploying on YARN in flip6 mode, the Flink binaries are not deleted from HDFS after the cluster shuts down.

Steps to reproduce

1. Submit job in YARN job mode, non-detached:

    HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster -yjm 2048 -ytm 2048 ./examples/streaming/WordCount.jar 

2. Check contents of /user/hadoop/.flink/<application_id> on HDFS after job is finished:

   [hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls /user/hadoop/.flink/application_1523966184826_0016
   Found 6 items
   -rw-r--r--   1 hadoop hadoop        583 2018-04-17 14:54 /user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
   -rw-r--r--   1 hadoop hadoop        332 2018-04-17 14:54 /user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
   -rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 /user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
   drwxrwxrwx   - hadoop hadoop          0 2018-04-17 14:54 /user/hadoop/.flink/application_1523966184826_0016/lib
   -rw-r--r--   1 hadoop hadoop       1939 2018-04-02 15:37 /user/hadoop/.flink/application_1523966184826_0016/log4j.properties
   -rw-r--r--   1 hadoop hadoop       2331 2018-04-02 15:37 /user/hadoop/.flink/application_1523966184826_0016/logback.xml
   

Issue Links

• links to: GitHub Pull Request #5938