Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3852

Optimize ContainerContext.isSuperSet to speed container reuse decisions

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Found an AM that was consuming high CPU. The stack trace below shows that container reuse compatibility check with a high number of local resources was the culprit.

      task scheduler compatibility check
      "DelayedContainerManager" #112 prio=5 os_prio=0 tid=0x0000000003b59800 nid=0x1edba runnable [0x00007fe13c232000]
         java.lang.Thread.State: RUNNABLE
      	at java.util.HashMap.putVal(HashMap.java:628)
      	at java.util.HashMap.putMapEntries(HashMap.java:514)
      	at java.util.HashMap.<init>(HashMap.java:489)
      	at org.apache.tez.dag.app.ContainerContext.localResourcesCompatible(ContainerContext.java:132)
      	at org.apache.tez.dag.app.ContainerContext.isSuperSet(ContainerContext.java:116)
      	at org.apache.tez.dag.app.rm.container.ContainerContextMatcher.isSuperSet(ContainerContextMatcher.java:50)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.canAssignTaskToContainer(YarnTaskSchedulerService.java:1543)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.getMatchingRequestWithoutPriority(YarnTaskSchedulerService.java:1492)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.access$500(YarnTaskSchedulerService.java:85)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService$NodeLocalContainerAssigner.assignReUsedContainer(YarnTaskSchedulerService.java:1870)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.assignReUsedContainerWithLocation(YarnTaskSchedulerService.java:1754)
      	- locked <0x00000006e0d12600> (a org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.assignReUsedContainersWithLocation(YarnTaskSchedulerService.java:1712)
      	- locked <0x00000006e0d12600> (a org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.tryAssignReUsedContainers(YarnTaskSchedulerService.java:578)
      	- locked <0x00000006e0d12600> (a org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.access$800(YarnTaskSchedulerService.java:85)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.doAssignAll(YarnTaskSchedulerService.java:2103)
      	- locked <0x00000006e0d12600> (a org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.mainLoop(YarnTaskSchedulerService.java:1984)
      	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.run(YarnTaskSchedulerService.java:1974)
      

        Attachments

        1. TEZ-3852.001.patch
          3 kB
          Jonathan Turner Eagles
        2. TEZ-3852.002.patch
          4 kB
          Jonathan Turner Eagles
        3. TEZ-3852.003.patch
          5 kB
          Jonathan Turner Eagles

          Activity

            People

            • Assignee:
              jeagles Jonathan Turner Eagles
              Reporter:
              jeagles Jonathan Turner Eagles
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: