[SOLR-6530] Commits under network partition can put any node in down state

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 4.10.2, 5.0, 6.0
• Component/s: SolrCloud
• Assignee: Shalin Shekhar Mangar
• Reporter: Shalin Shekhar Mangar
• Created: 17/Sep/14 15:27
• Updated: 09/May/16 18:58
• Resolved: 16/Oct/14 08:26

Description

Commits are executed by any node in SolrCloud i.e. they're not routed via the leader like other updates.

1. Suppose there's 1 collection, 1 shard, 2 replicas (A and B) and A is the leader
2. Suppose a commit request is made to node B during a time where B cannot talk to A due to a partition for any reason (failing switch, heavy GC, whatever)
3. B fails to distribute the commit to A (times out) and asks A to recover
4. This was okay earlier because a leader just ignores recovery requests but with leader initiated recovery code, B puts A in the "down" state and A can never get out of that state.

tl;dr; During network partitions, if enough commit/optimize requests are sent to the cluster, all the nodes in the cluster will eventually be marked as "down".

Issue Links

• is related to: SOLR-6536 Refactor DistributedUpdateProcessor's leader logic