[YARN-5288] Resource Localization fails due to leftover files

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9.0
• Component/s: nodemanager
• Assignee: Unassigned
• Reporter: Yufei Gu
• Created: 22/Jun/16 17:12
• Updated: 20/Mar/17 18:13

Description

NM restart didn't clean up all user cache. The leftover files can cause resource localization failure.

2016-06-14 23:09:12,717 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: 
java.io.IOException: Rename cannot overwrite non empty destination directory /data/5/yarn/nm/usercache/xxx/filecache/4567
        at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:716)
        at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236)
        at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:659)
        at org.apache.hadoop.fs.FileContext.rename(FileContext.java:912)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Issue Links

• is related to: YARN-3832 Resource Localization fails on a cluster due to existing cache directories