[YARN-10194] YARN RMWebServices /scheduler-conf/validate leaks ZK Connections

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0, 3.2.2, 3.4.0
• Component/s: capacityscheduler
• Target Version/s: 3.3.0
• Hadoop Flags: Reviewed
• Assignee: Prabhu Joseph
• Reporter: Akhil PB
• Created: 12/Mar/20 17:01
• Updated: 22/May/20 13:29
• Resolved: 28/Apr/20 15:53

Description

YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Validation API creates a new CapacityScheduler and missed to close after the validation. Every CapacityScheduler#init opens MutableCSConfigurationProvider which opens ZKConfigurationStore and creates a ZK Connection.

ZK LOGS

-03-12 16:45:51,881 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: [2 times] Error accepting new connection: Too many connections from /172.27.99.64 - max is 60
2020-03-12 16:45:52,449 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new connection: Too many connections from /172.27.99.64 - max is 60
2020-03-12 16:45:52,710 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new connection: Too many connections from /172.27.99.64 - max is 60
2020-03-12 16:45:52,876 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: [4 times] Error accepting new connection: Too many connections from /172.27.99.64 - max is 60
2020-03-12 16:45:53,068 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: [2 times] Error accepting new connection: Too many connections from /172.27.99.64 - max is 60
2020-03-12 16:45:53,391 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: [2 times] Error accepting new connection: Too many connections from /172.27.99.64 - max is 60
2020-03-12 16:45:54,008 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new connection: Too many connections from /172.27.99.64 - max is 60
2020-03-12 16:45:54,287 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new connection: Too many connections from /172.27.99.64 - max is 60
2020-03-12 16:45:54,483 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: [4 times] Error accepting new connection: Too many connections from /172.27.99.64 - max is 60

And there is an another bug in ZKConfigurationStore which has not handled close() of ZKCuratorManager.