Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10987

Solr Cloud overseer node becomes unreachable. Issue Started Recently

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Invalid
    • Affects Version/s: 6.1
    • Fix Version/s: None
    • Component/s: SolrCloud
    • Environment:

      Description

      We host a Solr Cloud of 5 Nodes for Solr Instances and 3 Zookeeper nodes to maintain the cloud. We have over 70 million docs spread across 13 collections with 40K more documents being added every day almost near time within spans of 5 to 6 minutes.

      The System was working as expected and as required for th elast 7 months until suddenly we saw the following exception and all of our instances went offline. We restarted the instances and the cloud ran smoothly for three days before it came crashing down again.

      Exception It gives before it goes down is as follows:

      3542285 ERROR (OverseerCollectionConfigSetProcessor-98221003671470081-prod-solr-node01:9080_solr-n_0000000106) [ ] o.a.s.c.OverseerTaskProcessor
      org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
      at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
      at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
      at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
      at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
      at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
      at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
      at java.lang.Thread.run(Unknown Source)

        Attachments

        1. solr.zip
          8 kB
          RAHAT BHALLA

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rbhalla RAHAT BHALLA
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: