[SOLR-7842] ZK connection loss or session expiry events should not fire config directory listeners

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 5.2.1
• Fix Version/s: 5.3, 6.0
• Component/s: SolrCloud
• Labels: difficulty-easy impact-low
• Assignee: Shalin Shekhar Mangar
• Reporter: Shalin Shekhar Mangar
• Created: 28/Jul/15 15:18
• Updated: 09/May/16 18:44
• Resolved: 29/Jul/15 08:57

Description

The watcher on the config directory has the following in the process method:

Stat stat = null;
      try {
        stat = zkClient.exists(zkDir, null, true);
      } catch (KeeperException e) {
        //ignore , it is not a big deal
      } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
      }

      boolean resetWatcher = false;
      try {
        resetWatcher = fireEventListeners(zkDir);
      } finally {
        if (Event.EventType.None.equals(event.getType())) {
          log.info("A node got unwatched for {}", zkDir);
        } else {
          if (resetWatcher) setConfWatcher(zkDir, this, stat);
          else log.info("A node got unwatched for {}", zkDir);
        }
      }

Even if the watcher is fired because of session expiry or connection loss, the fireEventListeners() method is executed and all subsequent listener invocations fail due to the loss of connection/session. All this is logged as well.

466879 WARN  (Thread-78) [   ] o.a.s.c.ZkController listener throws error
org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/jepsen/params.json
        at org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:158)
        at org.apache.solr.core.SolrConfig.refreshRequestParams(SolrConfig.java:909)
        at org.apache.solr.core.SolrCore$11.run(SolrCore.java:2585)
        at org.apache.solr.cloud.ZkController$4.run(ZkController.java:2385)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/jepsen/params.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:302)
        at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:299)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:299)
        at org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:148)
        ... 3 more

We should check the keeper state in addition to the event type and ignore such events.