Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-9196

YARN: Flink binaries are not deleted from HDFS after cluster shutdown

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.5.0
    • Component/s: Deployment / YARN
    • Labels:

      Description

      When deploying on YARN in flip6 mode, the Flink binaries are not deleted from HDFS after the cluster shuts down.

      Steps to reproduce

      1. Submit job in YARN job mode, non-detached:
         HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster -yjm 2048 -ytm 2048 ./examples/streaming/WordCount.jar 
      2. Check contents of /user/hadoop/.flink/<application_id> on HDFS after job is finished:
        [hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls /user/hadoop/.flink/application_1523966184826_0016
        Found 6 items
        -rw-r--r--   1 hadoop hadoop        583 2018-04-17 14:54 /user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
        -rw-r--r--   1 hadoop hadoop        332 2018-04-17 14:54 /user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
        -rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 /user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
        drwxrwxrwx   - hadoop hadoop          0 2018-04-17 14:54 /user/hadoop/.flink/application_1523966184826_0016/lib
        -rw-r--r--   1 hadoop hadoop       1939 2018-04-02 15:37 /user/hadoop/.flink/application_1523966184826_0016/log4j.properties
        -rw-r--r--   1 hadoop hadoop       2331 2018-04-02 15:37 /user/hadoop/.flink/application_1523966184826_0016/logback.xml
        

        Attachments

        1. 0001-xxx.patch
          2 kB
          yuqi

          Issue Links

            Activity

              People

              • Assignee:
                gjy Gary Yao
                Reporter:
                gjy Gary Yao
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: