[SOLR-16178] ZkController#fireEventListeners thread should be shutdown on close

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Kevin Risden
• Reporter: Kevin Risden
• Created: 03/May/22 17:41
• Updated: 12/May/22 00:35
• Resolved: 03/May/22 18:47

Description

ZkController#fireEventListeners creates a new thread but it is not stopped or interrupted on shutdown. This leaks threads.

I started looking into this because of TestBulkSchemaConcurrent and leaking threads:

   1) Thread[id=20840, name=ZKEventListenerThread, state=TIMED_WAITING, group=TGRP-TestBulkSchemaConcurrent]
        at java.base@17.0.2/java.lang.Thread.sleep(Native Method)
        at app//org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:161)
        at app//org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:82)
        at app//org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:318)
        at app//org.apache.solr.cloud.ZkController.setConfWatcher(ZkController.java:2777)
        at app//org.apache.solr.cloud.ZkController.getConfDirListeners(ZkController.java:2699)
        at app//org.apache.solr.cloud.ZkController.registerConfListenerForCore(ZkController.java:2679)
        at app//org.apache.solr.core.SolrCore.registerConfListener(SolrCore.java:3345)
        at app//org.apache.solr.core.SolrCore.<init>(SolrCore.java:1183)
        at app//org.apache.solr.core.SolrCore.reload(SolrCore.java:780)
        at app//org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1857)
        at app//org.apache.solr.core.SolrCore.lambda$getConfListener$21(SolrCore.java:3394)
        at app//org.apache.solr.core.SolrCore$$Lambda$926/0x0000000801430b18.run(Unknown Source)
        at app//org.apache.solr.cloud.ZkController.lambda$fireEventListeners$18(ZkController.java:2762)
        at app//org.apache.solr.cloud.ZkController$$Lambda$1401/0x0000000801768770.run(Unknown Source)
        at java.base@17.0.2/java.lang.Thread.run(Thread.java:833)

It turns out this thread leak happens a lot: https://lists.apache.org/list?[e-mail]:lte=1M:%22fireEventListeners%22

Issue Links

• is caused by: SOLR-6533 Support editing common solrconfig.xml values
• is related to: SOLR-16154 ZKEventListenerThread leaks from tests
• links to: GitHub Pull Request #833