[SOLR-12807] out of memory error due to a lot of zk watchers in solr cloud

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 6.1
• Fix Version/s: 6.5.1, 6.6, 7.0
• Component/s: SolrCloud
• Assignee: Unassigned
• Reporter: Mine Orange
• Created: 26/Sep/18 07:07
• Updated: 08/Jun/19 14:59
• Resolved: 28/Sep/18 07:43

Description

Analyzing the dump file,we found a lot of watchers in childWatches of ZKWatchManager,nearly 1.8G,the znode of childWatches is /overseer/collection-queue-work,confirm that it is not because of the frequent use of collection API and the network is normal. 

The instance is the overseer leader of a solr cluster and did not restart for more than a year,suspect that the watchers grow with time.

Our solr version is 6.1 and zookeeper version is 3.4.9.