[YARN-9714] ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.2.1
• Component/s: resourcemanager
• Labels: memory-leak
• Target Version/s: 3.2.1
• Hadoop Flags: Reviewed
• Assignee: Tao Yang
• Reporter: Tao Yang
• Created: 30/Jul/19 11:03
• Updated: 30/Aug/19 07:15
• Resolved: 30/Aug/19 05:12

Description

Recently RM full GC happened in one of our clusters, after investigating the dump memory and jstack, I found two places in RM may cause memory leaks after RM transitioned to standby:

1. Release cache cleanup timer in AbstractYarnScheduler never be canceled.
2. ZooKeeper connection in ZKRMStateStore never be closed.

To solve those leaks, we should close the connection or cancel the timer when services are stopping.