[YARN-6708] Nodemanager container crash after ext3 folder limit

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
• Hadoop Flags: Reviewed
• Assignee: Bibin Chundatt
• Reporter: Bibin Chundatt
• Created: 10/Jun/17 10:26
• Updated: 06/Jul/17 15:04
• Resolved: 06/Jul/17 14:56

Description

Configure umask as 027 for nodemanager service user
and yarn.nodemanager.local-cache.max-files-per-directory as 40. After 4 private dir localization next directory will be 0/14

Local Directory cache manager

vm2:/opt/hadoop/release/data/nmlocal/usercache/mapred/filecache # l
total 28
drwx--x--- 7 mapred hadoop 4096 Jun 10 14:35 ./
drwxr-s--- 4 mapred hadoop 4096 Jun 10 12:07 ../
drwxr-x--- 3 mapred users  4096 Jun 10 14:36 0/
drwxr-xr-x 3 mapred users  4096 Jun 10 12:15 10/
drwxr-xr-x 3 mapred users  4096 Jun 10 12:22 11/
drwxr-xr-x 3 mapred users  4096 Jun 10 12:27 12/
drwxr-xr-x 3 mapred users  4096 Jun 10 12:31 13/

drwxr-x--- 3 mapred users 4096 Jun 10 14:36 0/ is only 750
Nodemanager user will not be able check for localization path exists or not.

LocalResourcesTrackerImpl

    case REQUEST:
      if (rsrc != null && (!isResourcePresent(rsrc))) {
        LOG.info("Resource " + rsrc.getLocalPath()
            + " is missing, localizing it again");
        removeResource(req);
        rsrc = null;
      }
      if (null == rsrc) {
        rsrc = new LocalizedResource(req, dispatcher);
        localrsrc.put(req, rsrc);
      }
      break;

isResourcePresent will always return false and same resource will be localized to 0 to next unique number