[SOLR-17198] Affinity Placement Plugin can fail when getting metrics, if multiple replicas claim shard leadership

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 9.4
• Fix Version/s: 9.6
• Component/s: SolrCloud
• Assignee: Unassigned
• Reporter: Paul McArthur
• Created: 06/Mar/24 19:03
• Updated: 28/Apr/24 19:29
• Resolved: 16/Mar/24 05:03

Description

Using Solr 9.4 with 16 nodes, I observe that about 25% of our Split Shard requests are failing. The error is a RuntimeException that is raised by the AttributeFetcher as it compiles the metrics that will be used by the plugin.

 

The AttributeFetcher is making /admin/metrics requests to each node, and currently it expects to be able to establish a consistent  view of shard leadership across the cluster from the responses.

However, we see this exception:

 

Caused by: java.lang.RuntimeException: two replicas claim to be the shard leader! existing=org.apache.solr.cluster.placement.impl.CollectionMetricsBuilder$ReplicaMetricsBuilder@56e219b9 and current org.apache.solr.cluster.placement.impl.CollectionMetricsBuilder$ReplicaMetricsBuilder@406bcfd8
	at org.apache.solr.cluster.placement.impl.CollectionMetricsBuilder$ShardMetricsBuilder.lambda$build$0(CollectionMetricsBuilder.java:84)
	at java.base/java.util.HashMap.forEach(HashMap.java:1429)
	at org.apache.solr.cluster.placement.impl.CollectionMetricsBuilder$ShardMetricsBuilder.build(CollectionMetricsBuilder.java:76)
	at org.apache.solr.cluster.placement.impl.CollectionMetricsBuilder.lambda$build$0(CollectionMetricsBuilder.java:39)
	at java.base/java.util.HashMap.forEach(HashMap.java:1429)
	at org.apache.solr.cluster.placement.impl.CollectionMetricsBuilder.build(CollectionMetricsBuilder.java:39)
	at org.apache.solr.cluster.placement.impl.AttributeFetcherImpl.lambda$fetchAttributes$17(AttributeFetcherImpl.java:213)
	at java.base/java.util.HashMap.forEach(HashMap.java:1429)
	at org.apache.solr.cluster.placement.impl.AttributeFetcherImpl.fetchAttributes(AttributeFetcherImpl.java:212)
	at org.apache.solr.cluster.placement.plugins.AffinityPlacementFactory$AffinityPlacementPlugin.getBaseWeightedNodes(AffinityPlacementFactory.java:284)
	at org.apache.solr.cluster.placement.plugins.OrderedNodePlacementPlugin.getWeightedNodes(OrderedNodePlacementPlugin.java:311)
	at org.apache.solr.cluster.placement.plugins.OrderedNodePlacementPlugin.computePlacements(OrderedNodePlacementPlugin.java:85)
	at org.apache.solr.cluster.placement.impl.PlacementPluginAssignStrategy.assign(PlacementPluginAssignStrategy.java:84)
	at org.apache.solr.cloud.api.collections.Assign$AssignStrategy.assign(Assign.java:446)
	at org.apache.solr.cloud.api.collections.SplitShardCmd.split(SplitShardCmd.java:689) 

 

 

This indicates that more than one replica for a given Shard has responded with leader=true in the replica metrics.

I think there are legitimate reasons this can occur:

1. It may be fundamentally impossible to always be able to build a consistent view of shard leadership from querying a set of distributed nodes
2. /admin/metrics requests are sent sequentially to each node in turn. It is possible that shard leadership may change between making the request to different nodes that host replicas for a shard

Issue Links

• links to: GitHub Pull Request #2335