[YARN-3832] Resource Localization fails on a cluster due to existing cache directories

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.6.1, 2.8.0, 2.7.1, 3.0.0-alpha1
• Component/s: nodemanager
• Labels: 2.6.1-candidate
• Hadoop Flags: Reviewed
• Assignee: Brahma Reddy Battula
• Reporter: Ranga Swamy
• Created: 19/Jun/15 13:31
• Updated: 06/Jan/17 00:51
• Resolved: 24/Jun/15 16:39

Description

We have found resource localization fails on a cluster with following error.

Got this error in hadoop-2.7.0 release which was fixed in 2.6.0 (YARN-2624)

Application application_1434703279149_0057 failed 2 times due to AM Container for appattempt_1434703279149_0057_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://S0559LDPag68:45020/cluster/app/application_1434703279149_0057Then, click on links to logs of each attempt.
Diagnostics: Rename cannot overwrite non empty destination directory /opt/hdfsdata/HA/nmlocal/usercache/root/filecache/39
java.io.IOException: Rename cannot overwrite non empty destination directory /opt/hdfsdata/HA/nmlocal/usercache/root/filecache/39
at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735)
at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:244)
at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678)
at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Failing this attempt. Failing the application.

Issue Links

• relates to: YARN-5288 Resource Localization fails due to leftover files