Details
Minor Description
We found an issue while performing stability tests on SolrCloud. Under certain circumstances, a node will get in an endless loop trying to recover. I've seen this happen in a two node setup, by following these steps:
1) Node A started
2) Node B started
3) Node B stopped
4) Node B started, and immediately Node A stopped (normal graceful shutdown).
At this point node B will throw connection refused messages while trying to sync to node A. For some reason (not always) this leads to a corrupt state where node B enters an infinite loop trying to recover from node A (it still thinks the cluster has two nodes). I think the leader election process started just fine, but since recovery is running async, at some point node B published it state as recovery failed, hence causing leader election to fail.
Zookeeper /live_nodes has only one file.
This shows on the logs:
0:57:18,960 INFO INFO [ShardLeaderElectionContext] (main-EventThread) Running the leader process.
10:57:19,068 INFO INFO [ShardLeaderElectionContext] (main-EventThread) Checking if I should try and be the leader.
10:57:19,068 INFO INFO [ShardLeaderElectionContext] (main-EventThread) My last published State was recovery_failed, I won't be the leader.
10:57:19,068 INFO INFO [ShardLeaderElectionContext] (main-EventThread) There may be a better leader candidate than us - going back into recovery
10:57:19,118 INFO INFO [DefaultSolrCoreState] (main-EventThread) Running recovery - first canceling any ongoing recovery
10:57:19,118 WARN WARN [RecoveryStrategy] (main-EventThread) Stopping recovery for zkNodeName=10.50.100.30:8998_solr_myCollectioncore=myCollection
10:57:19,869 ERROR ERROR [RecoveryStrategy] (RecoveryThread) Error while trying to recover. core=myCollection:org.apache.solr.common.SolrException: No registered leader was found, collection:myCollection slice:shard1
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:467)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:321)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
10:57:19,869 ERROR ERROR [RecoveryStrategy] (RecoveryThread) Recovery failed - trying again... (0) core=myCollection
10:57:19,869 ERROR ERROR [RecoveryStrategy] (RecoveryThread) Recovery failed - interrupted. core=myCollection
10:57:19,869 ERROR ERROR [RecoveryStrategy] (RecoveryThread) Recovery failed - I give up. core=myCollection
10:57:19,869 INFO INFO [ZkController] (RecoveryThread) publishing core=myCollection state=recovery_failed
10:57:19,869 INFO INFO [ZkController] (RecoveryThread) numShards not found on descriptor - reading it from system property
10:57:19,902 WARN WARN [RecoveryStrategy] (RecoveryThread) Stopping recovery for zkNodeName=10.50.100.30:8998_solr_myCollectioncore=myCollection
10:57:19,902 INFO INFO [RecoveryStrategy] (RecoveryThread) Finished recovery process. core=myCollection
10:57:19,902 INFO INFO [RecoveryStrategy] (RecoveryThread) Starting recovery process. core=myCollection recoveringAfterStartup=false
Solr Version: 4.2.1.2013.03.26.08.26.55
Other references to the same issue: