Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
Currently ShuffleHandler and LocalDirsHandlerService share AllocatorPerContext object in LocalDirAllocator for configuration NM_LOCAL_DIRS because AllocatorPerContext are stored in a static TreeMap with configuration name as key
private static Map <String, AllocatorPerContext> contexts = new TreeMap<String, AllocatorPerContext>();
LocalDirsHandlerService and ShuffleHandler both create a LocalDirAllocator using NM_LOCAL_DIRS. Even they don't use the same Configuration object, but they will use the same AllocatorPerContext object. Also LocalDirsHandlerService may change NM_LOCAL_DIRS value in its Configuration object to exclude full and bad local dirs, ShuffleHandler always uses the original NM_LOCAL_DIRS value in its Configuration object. So every time AllocatorPerContext#confChanged is called by ShuffleHandler after LocalDirsHandlerService, AllocatorPerContext need be reinitialized because NM_LOCAL_DIRS value is changed. This will cause some overhead.
String newLocalDirs = conf.get(contextCfgItemName); if (!newLocalDirs.equals(savedLocalDirs)) {
So it will be a good improvement to not share the same AllocatorPerContext instance between ShuffleHandler and LocalDirsHandlerService.
Attachments
Attachments
Issue Links
- is related to
-
YARN-6277 Nodemanager heap memory leak
- Resolved