[YARN-2624] Resource Localization fails on a cluster due to existing cache directories

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.0, 2.5.1
• Fix Version/s: 2.6.0
• Component/s: nodemanager
• Hadoop Flags: Reviewed
• Assignee: Anubhav Dhoot
• Reporter: Anubhav Dhoot
• Created: 30/Sep/14 00:32
• Updated: 25/Oct/19 20:24
• Resolved: 02/Oct/14 17:46

Description

We have found resource localization fails on a cluster with following error in certain cases.

INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Failed to download rsrc { { hdfs://<blahhostname>:8020/tmp/hive-hive/hive_2014-09-29_14-55-45_184_6531377394813896912-12/-mr-10004/95a07b90-2448-48fc-bcda-cdb7400b4975/map.xml, 1412027745352, FILE, null },pending,[(container_1411670948067_0009_02_000001)],443533288192637,DOWNLOADING}
java.io.IOException: Rename cannot overwrite non empty destination directory /data/yarn/nm/filecache/27
	at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:716)
	at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:228)
	at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:659)
	at org.apache.hadoop.fs.FileContext.rename(FileContext.java:906)
	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366)
	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:59)

Issue Links

• is related to: MAPREDUCE-6441 Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes