[SOLR-6837] Inconsistent replicas when update is succesful against leader partitioned from all replicas

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.10.2
• Component/s: SolrCloud
• Labels: difficulty-hard impact-medium
• Assignee: Unassigned
• Reporter: Shalin Shekhar Mangar
• Created: 11/Dec/14 20:12
• Updated: 11/Dec/14 20:14

Description

Refer to the following question on solr-user:
https://www.marshut.net/kttiuz/inconsistent-doc-value-across-two-nodes-very-simple-test-what-s-the-expected-behavior.html

Config
Solr 4.7.2 / Jetty.
SoldCloud on two nodes, and 3 ZK, all running in localhost.
single collection: single shard with two replicas.

Reproducing:
start node1 9.148.58.114:8983
start node2 9.148.58.114:8984
Cluster state: node1 leader. node2 active.

index value 'A' (id="change me").
query and expect 'A' -> success

Stop node2
Cluster state: node1 leader. node2 gone.
query and expect 'A' -> success

Update document value from 'A'->'B'
query and expect 'B' -> success

Stop node1
then
Start node2.
Cluster state: node1 gone. node2 down.

104510 [coreZkRegister-1-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=5.27665925E14ms

wait 3m.

184679 [coreZkRegister-1-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext I am the new leader: http://9.148.58.114:8984/solr/quick-results-collection_shard1_replica2/ shard1
Cluster state: node1 gone. node2 leader.

query and expect 'A' (old value) -> success

start node1
Cluster state: node1 actove. node2 leader.

Inconsistency:
Querying node1 always returns 'B'. http://localhost:8983/solr/quick-results-collection_shard1_replica1/select?q=*%3A*&wt=json&indent=true
Querying node1 always returns 'A'. http://localhost:8984/solr/quick-results-collection_shard1_replica2/select?q=*%3A*&wt=json&indent=true

In such a case, the final steady state of the system has inconsistent replicas.