[SOLR-4260] Inconsistent numDocs between leader and replica

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 4.6.1, 6.0
• Component/s: SolrCloud

Environment

5.0.0.2013.01.04.15.31.51

• Assignee: Mark Miller
• Reporter: Markus Jelsma
• Created: 04/Jan/13 17:53
• Updated: 09/May/16 18:51
• Resolved: 02/Feb/14 15:19

Description

After wiping all cores and reindexing some 3.3 million docs from Nutch using CloudSolrServer we see inconsistencies between the leader and replica for some shards.

Each core hold about 3.3k documents. For some reason 5 out of 10 shards have a small deviation in then number of documents. The leader and slave deviate for roughly 10-20 documents, not more.

Results hopping ranks in the result set for identical queries got my attention, there were small IDF differences for exactly the same record causing a record to shift positions in the result set. During those tests no records were indexed. Consecutive catch all queries also return different number of numDocs.

We're running a 10 node test cluster with 10 shards and a replication factor of two and frequently reindex using a fresh build from trunk. I've not seen this issue for quite some time until a few days ago.

Issue Links

• is related to: SOLR-5625 Add to testing for SolrCmdDistributor
• is related to: SOLR-5643 ConcurrentUpdateSolrServer will sometimes not spawn a new Runner thread even though there are updates in the queue.
• relates to: SOLR-5397 Replication can fail silently in some cases.
• relates to: SOLR-5552 Leader recovery process can select the wrong leader if all replicas for a shard are down and trying to recover as well as lose updates that should have been recovered.
• relates to: SOLR-5573 ChaosMonkey should randomly turn off Solr's commit on shutdown option.