[TEZ-3852] Optimize ContainerContext.isSuperSet to speed container reuse decisions

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.1
• Hadoop Flags: Reviewed
• Assignee: Jonathan Turner Eagles
• Reporter: Jonathan Turner Eagles
• Created: 10/Oct/17 22:22
• Updated: 05/Jan/18 00:17
• Resolved: 12/Oct/17 14:35

Description

Found an AM that was consuming high CPU. The stack trace below shows that container reuse compatibility check with a high number of local resources was the culprit.

task scheduler compatibility check

"DelayedContainerManager" #112 prio=5 os_prio=0 tid=0x0000000003b59800 nid=0x1edba runnable [0x00007fe13c232000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.putVal(HashMap.java:628)
	at java.util.HashMap.putMapEntries(HashMap.java:514)
	at java.util.HashMap.<init>(HashMap.java:489)
	at org.apache.tez.dag.app.ContainerContext.localResourcesCompatible(ContainerContext.java:132)
	at org.apache.tez.dag.app.ContainerContext.isSuperSet(ContainerContext.java:116)
	at org.apache.tez.dag.app.rm.container.ContainerContextMatcher.isSuperSet(ContainerContextMatcher.java:50)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.canAssignTaskToContainer(YarnTaskSchedulerService.java:1543)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.getMatchingRequestWithoutPriority(YarnTaskSchedulerService.java:1492)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.access$500(YarnTaskSchedulerService.java:85)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService$NodeLocalContainerAssigner.assignReUsedContainer(YarnTaskSchedulerService.java:1870)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.assignReUsedContainerWithLocation(YarnTaskSchedulerService.java:1754)
	- locked <0x00000006e0d12600> (a org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.assignReUsedContainersWithLocation(YarnTaskSchedulerService.java:1712)
	- locked <0x00000006e0d12600> (a org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.tryAssignReUsedContainers(YarnTaskSchedulerService.java:578)
	- locked <0x00000006e0d12600> (a org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService.access$800(YarnTaskSchedulerService.java:85)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.doAssignAll(YarnTaskSchedulerService.java:2103)
	- locked <0x00000006e0d12600> (a org.apache.tez.dag.app.rm.YarnTaskSchedulerService)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.mainLoop(YarnTaskSchedulerService.java:1984)
	at org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.run(YarnTaskSchedulerService.java:1974)