[YARN-5296] NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.0
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: nodemanager
• Target Version/s: 2.9.0
• Hadoop Flags: Reviewed
• Assignee: Junping Du
• Reporter: Karam Singh
• Created: 27/Jun/16 16:21
• Updated: 25/Oct/19 20:26
• Resolved: 06/Jul/16 01:07

Description

Ran tests in following manner,
1. Run GridMix of 768 sequestionally around 17 times to execute about 12.9K apps.
2. After 4-5hrs take Check NM Heap using Memory Analyser. It report around 96% Heap is being used my ContainerMetrics
3. Run 7 more GridMix run for have around 18.2apps ran in total. Again check NM heap using Memory Analyser again 96% heap is being used by ContainerMetrics.
4. Start one more grimdmix run, while run going on , NMs started going down with OOM, around running 18.7K+, On analysing NM heap using Memory analyser, OOM was caused by ContainerMetrics

Issue Links

• relates to: YARN-5190 Registering/unregistering container metrics triggered by ContainerEvent and ContainersMonitorEvent are conflict which cause uncaught exception in ContainerMonitorImpl
• relates to: HADOOP-13362 DefaultMetricsSystem leaks the source name when a source unregisters