Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5345

IOManager failed to properly clean up temp file directory

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.3
    • Fix Version/s: 1.2.0, 1.3.0
    • Component/s: None
    • Labels:

      Description

      While testing 1.1.3 RC3, I have the following message in my log:

      2016-12-15 14:46:05,450 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - Timer service is shutting down.
      2016-12-15 14:46:05,452 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: control events generator (29/40) (73915a232ba09e642f9dff92f8c8773a) switched from CANCELING to CANCELED.
      2016-12-15 14:46:05,452 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Source: control events generator (29/40) (73915a232ba09e642f9dff92f8c8773a).
      2016-12-15 14:46:05,454 INFO  org.apache.flink.yarn.YarnTaskManager                         - Un-registering task and sending final execution state CANCELED to JobManager for task Source: control events genera
      tor (73915a232ba09e642f9dff92f8c8773a)
      2016-12-15 14:46:40,609 INFO  org.apache.flink.yarn.YarnTaskManagerRunner                   - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
      2016-12-15 14:46:40,611 INFO  org.apache.flink.runtime.blob.BlobCache                       - Shutting down BlobCache
      2016-12-15 14:46:40,724 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.240.0.34:33635] has failed, address is now gated for [5000] ms.
       Reason is: [Disassociated].
      2016-12-15 14:46:40,808 ERROR org.apache.flink.runtime.io.disk.iomanager.IOManager          - IOManager failed to properly clean up temp file directory: /yarn/nm/usercache/robert/appcache/application_148129128
      9979_0024/flink-io-f0ff3f66-b9e2-4560-881f-2ab43bc448b5
      java.lang.IllegalArgumentException: /yarn/nm/usercache/robert/appcache/application_1481291289979_0024/flink-io-f0ff3f66-b9e2-4560-881f-2ab43bc448b5/62e14e1891fe1e334c921dfd19a32a84/StreamMap_11_24/dummy_state does not exist
              at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1637)
              at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
              at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
              at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
              at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
              at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
              at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
              at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
              at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
              at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
              at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
              at org.apache.flink.runtime.io.disk.iomanager.IOManager.shutdown(IOManager.java:109)
              at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.shutdown(IOManagerAsync.java:185)
              at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$1.run(IOManagerAsync.java:105)
      
      

      This was the last message logged from that machine. I suspect two threads are trying to clean up the directories during shutdown?

        Attachments

          Activity

            People

            • Assignee:
              StephanEwen Stephan Ewen
              Reporter:
              rmetzger Robert Metzger
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: