[YARN-4563] ContainerMetrics deadlocks

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Component/s: nodemanager

Environment

HDP 2.3.2 (Hadoop 2.7.1 + patches)

• Assignee: Unassigned
• Reporter: Akira Ajisaka
• Created: 08/Jan/16 10:19
• Updated: 24/Jan/19 10:37
• Resolved: 12/Jan/16 05:42

Description

On one of our environment, some NodeManagers' webapp do not working. I found a dead lock in the thread dump.

Found one Java-level deadlock:
=============================
"1193752357@qtp-907815246-22238":
  waiting to lock monitor 0x0000000005e20a18 (object 0x00000000f6afa048, a org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics),
  which is held by "2107307914@qtp-907815246-19994"
"2107307914@qtp-907815246-19994":
  waiting to lock monitor 0x0000000001a000a8 (object 0x00000000d4f1e1f8, a org.apache.hadoop.metrics2.impl.MetricsSystemImpl),
  which is held by "Timer for 'NodeManager' metrics system"
"Timer for 'NodeManager' metrics system":
  waiting to lock monitor 0x00000000027ade88 (object 0x00000000f6582df0, a org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics),
  which is held by "1530638165@qtp-907815246-19992"
"1530638165@qtp-907815246-19992":
  waiting to lock monitor 0x0000000001a000a8 (object 0x00000000d4f1e1f8, a org.apache.hadoop.metrics2.impl.MetricsSystemImpl),
  which is held by "Timer for 'NodeManager' metrics system"

Issue Links

• Is contained by: YARN-3619 ContainerMetrics unregisters during getMetrics and leads to ConcurrentModificationException
• links to: GitHub Pull Request #72