Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9508

YarnConfiguration areNodeLabel enabled is costly in allocation flow

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, 3.2.1
    • Component/s: None
    • Labels:
      None

      Description

      For every allocate request locking can be avoided. Improving performance

      "pool-6-thread-300" #624 prio=5 os_prio=0 tid=0x00007f2f91152800 nid=0x8ec5 waiting for monitor entry [0x00007f1ec6a8d000]
       java.lang.Thread.State: BLOCKED (on object monitor)
       at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2841)
       - waiting to lock <0x00007f1f8107c748> (a org.apache.hadoop.yarn.conf.YarnConfiguration)
       at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214)
       at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1268)
       at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1674)
       at org.apache.hadoop.yarn.conf.YarnConfiguration.areNodeLabelsEnabled(YarnConfiguration.java:3646)
       at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
       at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:274)
       at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:261)
       at org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:242)
       at org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
       at org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
       at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:427)
       - locked <0x00007f24dd3f9e40> (a org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService$AllocateResponseLock)
       at org.apache.hadoop.yarn.sls.appmaster.MRAMSimulator$1.run(MRAMSimulator.java:352)
       at org.apache.hadoop.yarn.sls.appmaster.MRAMSimulator$1.run(MRAMSimulator.java:349)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:422)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
       at org.apache.hadoop.yarn.sls.appmaster.MRAMSimulator.sendContainerRequest(MRAMSimulator.java:348)
       at org.apache.hadoop.yarn.sls.appmaster.AMSimulator.middleStep(AMSimulator.java:212)
       at org.apache.hadoop.yarn.sls.scheduler.TaskRunner$Task.run(TaskRunner.java:94)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        1. YARN-9508-003.patch
          11 kB
          Bilwa S T
        2. YARN-9508-002.patch
          11 kB
          Bilwa S T
        3. YARN-9508-001.patch
          10 kB
          Bilwa S T

          Activity

            People

            • Assignee:
              BilwaST Bilwa S T
              Reporter:
              bibinchundatt Bibin Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: