Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10506

Possible memory leak upon collection reload

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.5
    • Fix Version/s: 7.0
    • Component/s: Server
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Upon manual Solr Collection reloading, references to the closed SolrCore are not fully removed by the garbage collector as a strong reference to the ZkIndexSchemaReader is held in a ZooKeeper Watcher that watches for schema changes.

      In our case, this leads to a massive memory leak as managed resources are still referenced by the closed SolrCore. Our Solr cloud environment utilizes rather large managed resources (synonyms, stopwords). To reproduce, we fired out environment up and reloaded the collection 13 times. As a result we fully exhausted our heap. A closer look with the Yourkit profiler revealed 13 SolrCore instances, still holding strong references to the garbage collection root (see screenshot 1).

      Each SolrCore instance holds a single path with strong references to the gc root via a `Watcher` in `ZkIndexSchemaReader` (see screenshot 2). The ZkIndexSchemaReader registers a close hook in the SolrCore but the Zookeeper is not removed upon core close.

      We supplied a Github Pull Request (https://github.com/apache/lucene-solr/pull/197) that extracts the zookeeper `Watcher` as a static inner class. To eliminate the memory leak, the schema reader is held inside a `WeakReference` and the reference is explicitly removed on core close.

      Initially I wanted to supply a test case but unfortunately did not find a good starting point ...

        Attachments

        1. SOLR-10506.patch
          9 kB
          Christine Poerschke
        2. solr_gc_path_via_zk_WatchManager.png
          240 kB
          Torsten Bøgh Köster
        3. solr_collection_reload_13_cores.png
          186 kB
          Torsten Bøgh Köster

          Issue Links

            Activity

              People

              • Assignee:
                cpoerschke Christine Poerschke
                Reporter:
                tboeghk Torsten Bøgh Köster
              • Votes:
                4 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: