Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4095

Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently ShuffleHandler and LocalDirsHandlerService share AllocatorPerContext object in LocalDirAllocator for configuration NM_LOCAL_DIRS because AllocatorPerContext are stored in a static TreeMap with configuration name as key

        private static Map <String, AllocatorPerContext> contexts = 
                       new TreeMap<String, AllocatorPerContext>();
      

      LocalDirsHandlerService and ShuffleHandler both create a LocalDirAllocator using NM_LOCAL_DIRS. Even they don't use the same Configuration object, but they will use the same AllocatorPerContext object. Also LocalDirsHandlerService may change NM_LOCAL_DIRS value in its Configuration object to exclude full and bad local dirs, ShuffleHandler always uses the original NM_LOCAL_DIRS value in its Configuration object. So every time AllocatorPerContext#confChanged is called by ShuffleHandler after LocalDirsHandlerService, AllocatorPerContext need be reinitialized because NM_LOCAL_DIRS value is changed. This will cause some overhead.

            String newLocalDirs = conf.get(contextCfgItemName);
            if (!newLocalDirs.equals(savedLocalDirs)) {
      

      So it will be a good improvement to not share the same AllocatorPerContext instance between ShuffleHandler and LocalDirsHandlerService.

        Attachments

        1. YARN-4095.001.patch
          5 kB
          zhihai xu
        2. YARN-4095.000.patch
          7 kB
          zhihai xu

          Issue Links

            Activity

              People

              • Assignee:
                zxu zhihai xu
                Reporter:
                zxu zhihai xu
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: