[SOLR-15288] PRS replicas stay ACTIVE after a new node is stopped

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 8.8.1
• Fix Version/s: 8.8.2
• Assignee: Ishan Chattopadhyaya
• Reporter: Ishan Chattopadhyaya
• Created: 25/Mar/21 15:03
• Updated: 12/Apr/21 20:18
• Resolved: 12/Apr/21 20:08

Description

Steps to reproduce:
https://issues.apache.org/jira/browse/SOLR-15288?focusedCommentId=17312808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312808

---- Older description (that refers to a bug when started multiple nodes starting in same dir, SOLR-15305 deals with that issue now —
After a PRS collection is created using a single node cluster, and a new node is added and a replica for that collection is placed on the new node, restarting that new node causes problems with replica states.

Reproduce script:

# Start a fresh ZK on 2181
# docker container prune -f && docker run -it -p 2181:2181 --name=zk1 -h zk1 zookeeper:3.5.6

rm -rf server/logs/*
bin/solr stop -all
rm -rf server/solr/mycoll_shard1_replica_n1/ server/solr/mycoll_shard1_replica_n3/

bin/solr -c -p 9000 -z localhost:2181
curl "http://localhost:9000/solr/admin/collections?action=CREATE&name=mycoll&numShards=1&perReplicaState=true"
bin/solr -c -p 9001 -z localhost:2181
curl "http://localhost:9000/solr/admin/collections?action=ADDREPLICA&collection=mycoll&shard=shard1"
bin/solr stop -p 9001
bin/solr -c -p 9001 -z localhost:2181

Two problems:
1. Now look at the two replicas, both are down.
2. Also, as hitesh.khamesra found out, the second replica stays ACTIVE (not DOWN) after the second node (9001) is stopped.

Issue Links

• links to: GitHub Pull Request #71
• links to: GitHub Pull Request #2479