[SOLR-17257] Improve replica placement performance at high collection count

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 9.4
• Component/s: SolrCloud
• Labels: performance
• Assignee: Unassigned
• Reporter: Yohann Callea
• Created: 25/Apr/24 15:11
• Updated: 01/May/24 20:18
• Resolved: 01/May/24 19:54

Description

When operating Solr 9 with a significant number of collections (tens of thousands), we are currently facing performance issues when positioning new replicas using either the Minimize Cores or the Affinity placement strategy.

At this scale, each replica placement operation takes several seconds to complete and occupy gigabytes of heap, growing as the number of collections increase.  

To address these issues, we need to look for opportunities to rework AttributeFetcherImpl and the construction of SolrClientNodeStateProvider to eventually :

• Avoid gathering information about the whole cluster, including all collections and all replicas, when it is not absolutely necessary
• No longer have an implementation which time complexity depends on the number of nodes

Issue Links

• links to: GitHub Pull Request #2420