[SOLR-5821] Search inconsistency on SolrCloud replicas

• Type: Bug
• Status: Reopened
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 4.6.1, 4.7.1
• Component/s: SolrCloud
• Labels: cloud inconsistency replica search

Environment

SolrCloud:
1 shard, 2 replicas

Both instances/replicas have identical hardware/software:
CPU(s): 4
RAM: 8Gb
HDD: 100Gb
OS: CentOS 6.5
ZooKeeper 3.4.5
Tomcat 8.0.3
Solr 4.6.1

Servers are utilized to run Solr only.

• Assignee: Unassigned
• Reporter: Maxim Novikov
• Created: 06/Mar/14 18:24
• Updated: 05/Dec/14 16:57

Description

We use the following infrastructure:

SolrCloud with 1 shard and 2 replicas. The index is built using DataImportHandler (importing data from the database). The number of items in the index can vary from 100 to 100,000,000.

After indexing part of the data (not necessarily all the data, it is enough to have a small number of items in the search index), we can observe that Solr instances (replicas) return different results for the same search queries. I believe it happens because some of the results have the same scores, and Solr instances return those in a random order.

PS This is a critical issue for us as we use a load balancer to scale Solr through replicas, and as a result of this issue, we retrieve various results for the same queries all the time. They are not necessarily completely different, but even a couple of items that differ is a deal breaker.

The expected behaviour would be to always get identical results for the same search queries from all replicas. Otherwise, this "cloud" thing works just unreliably.