Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
-
None
-
Reviewed
Description
Configure umask as 027 for nodemanager service user
and yarn.nodemanager.local-cache.max-files-per-directory as 40. After 4 private dir localization next directory will be 0/14
Local Directory cache manager
vm2:/opt/hadoop/release/data/nmlocal/usercache/mapred/filecache # l total 28 drwx--x--- 7 mapred hadoop 4096 Jun 10 14:35 ./ drwxr-s--- 4 mapred hadoop 4096 Jun 10 12:07 ../ drwxr-x--- 3 mapred users 4096 Jun 10 14:36 0/ drwxr-xr-x 3 mapred users 4096 Jun 10 12:15 10/ drwxr-xr-x 3 mapred users 4096 Jun 10 12:22 11/ drwxr-xr-x 3 mapred users 4096 Jun 10 12:27 12/ drwxr-xr-x 3 mapred users 4096 Jun 10 12:31 13/
drwxr-x--- 3 mapred users 4096 Jun 10 14:36 0/ is only 750
Nodemanager user will not be able check for localization path exists or not.
LocalResourcesTrackerImpl
case REQUEST: if (rsrc != null && (!isResourcePresent(rsrc))) { LOG.info("Resource " + rsrc.getLocalPath() + " is missing, localizing it again"); removeResource(req); rsrc = null; } if (null == rsrc) { rsrc = new LocalizedResource(req, dispatcher); localrsrc.put(req, rsrc); } break;
isResourcePresent will always return false and same resource will be localized to 0 to next unique number