[YARN-4095] Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: nodemanager
• Hadoop Flags: Reviewed
• Assignee: Zhihai Xu
• Reporter: Zhihai Xu
• Created: 31/Aug/15 02:38
• Updated: 07/Apr/17 03:50
• Resolved: 23/Sep/15 15:45

Description

Currently ShuffleHandler and LocalDirsHandlerService share AllocatorPerContext object in LocalDirAllocator for configuration NM_LOCAL_DIRS because AllocatorPerContext are stored in a static TreeMap with configuration name as key

  private static Map <String, AllocatorPerContext> contexts = 
                 new TreeMap<String, AllocatorPerContext>();

LocalDirsHandlerService and ShuffleHandler both create a LocalDirAllocator using NM_LOCAL_DIRS. Even they don't use the same Configuration object, but they will use the same AllocatorPerContext object. Also LocalDirsHandlerService may change NM_LOCAL_DIRS value in its Configuration object to exclude full and bad local dirs, ShuffleHandler always uses the original NM_LOCAL_DIRS value in its Configuration object. So every time AllocatorPerContext#confChanged is called by ShuffleHandler after LocalDirsHandlerService, AllocatorPerContext need be reinitialized because NM_LOCAL_DIRS value is changed. This will cause some overhead.

      String newLocalDirs = conf.get(contextCfgItemName);
      if (!newLocalDirs.equals(savedLocalDirs)) {

So it will be a good improvement to not share the same AllocatorPerContext instance between ShuffleHandler and LocalDirsHandlerService.

Issue Links

• is related to: YARN-6277 Nodemanager heap memory leak