Upon manual Solr Collection reloading, references to the closed SolrCore are not fully removed by the garbage collector as a strong reference to the ZkIndexSchemaReader is held in a ZooKeeper Watcher that watches for schema changes.
In our case, this leads to a massive memory leak as managed resources are still referenced by the closed SolrCore. Our Solr cloud environment utilizes rather large managed resources (synonyms, stopwords). To reproduce, we fired out environment up and reloaded the collection 13 times. As a result we fully exhausted our heap. A closer look with the Yourkit profiler revealed 13 SolrCore instances, still holding strong references to the garbage collection root (see screenshot 1).
Each SolrCore instance holds a single path with strong references to the gc root via a `Watcher` in `ZkIndexSchemaReader` (see screenshot 2). The ZkIndexSchemaReader registers a close hook in the SolrCore but the Zookeeper is not removed upon core close.
We supplied a Github Pull Request (https://github.com/apache/lucene-solr/pull/197) that extracts the zookeeper `Watcher` as a static inner class. To eliminate the memory leak, the schema reader is held inside a `WeakReference` and the reference is explicitly removed on core close.
Initially I wanted to supply a test case but unfortunately did not find a good starting point ...