[SOLR-15028] SolrCloud shows cluster still healthy without failover even the node data directory is deleted

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 7.4
• Component/s: SolrCloud
• Assignee: Unassigned
• Reporter: Amy Bai
• Created: 03/Dec/20 02:54
• Updated: 13/Feb/21 00:09

Description

I found that SolrCloud won't check the IO status if the SolrCloud process is alive.

 

e.g. If I delete the data directory for one of the SolrCloud node, there are no errors report, and I can still log in to the SolrCloud Admin UI to create/query collections. SolrCloud Admin UI shows the collections' status is green.

Then, index/search queries keep failing because one of the node data directories is gone, but the node is not marked as down.

The replicas on the failed node are not working, but the Index/search queries didn't failover to other healthy replicas.  

 

The ERROR message as below shows:

"""

curl -X POST -H 'Content-Type: application/json' 'http://localhost:18983/solr/demo.public.test/update/json/docs' --data-binary ' { "a": "1", }' { "responseHeader":

{ "status":400, "QTime":6}

, "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","java.nio.file.NoSuchFileException"], "msg":"Error inserting document: ", "code":400}}

"""