[YARN-3727] For better error recovery, check if the directory exists before using it for localization.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0, 2.7.2, 2.6.2, 3.0.0-alpha1
• Component/s: nodemanager
• Target Version/s: 2.6.2
• Hadoop Flags: Reviewed
• Assignee: Zhihai Xu
• Reporter: Zhihai Xu
• Created: 28/May/15 01:53
• Updated: 02/Dec/16 00:31
• Resolved: 30/Sep/15 15:08

Description

For better error recovery, check if the directory exists before using it for localization.
We saw the following localization failure happened due to existing cache directories.

2015-05-11 18:59:59,756 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: DEBUG: FAILED { hdfs://XXXX/XXXXX/libjars/1234.jar, 1431395961545, FILE, null }, Rename cannot overwrite non empty destination directory /XXXX/8/yarn/nm/usercache/XXXX/filecache/21637
2015-05-11 18:59:59,756 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://XXXX/XXXXX/libjars/1234.jar(->/XXXX/8/yarn/nm/usercache/XXXX/filecache/21637/1234.jar) transitioned from DOWNLOADING to FAILED

The real cause for this failure may be disk failure, LevelDB operation failure for startResourceLocalization/finishResourceLocalization or others.

I wonder whether we can add error recovery code to avoid the localization failure by not using the existing cache directories for localization.

The exception happened at files.rename(dst_work, destDirPath, Rename.OVERWRITE) in FSDownload#call. Based on the following code, after the exception, the existing cache directory used by LocalizedResource will be deleted.

try {
     .........
      files.rename(dst_work, destDirPath, Rename.OVERWRITE);
    } catch (Exception e) {
      try {
        files.delete(destDirPath, true);
      } catch (IOException ignore) {
      }
      throw e;
    } finally {

Since the conflicting local directory will be deleted after localization failure,
I think it will be better to check if the directory exists before using it for localization to avoid the localization failure.