[SOLR-10726] SolrCloud opens multiple searchers on replica creation/startup

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 6.5.1
• Component/s: search, SolrCloud
• Labels: difficulty-medium impact-high
• Assignee: Unassigned
• Reporter: Shalin Shekhar Mangar
• Created: 22/May/17 19:12
• Updated: 08/Jun/19 15:19

Description

I was investigating some curious behavior reported by a customer around first searcher event listeners and multiple searchers being opened when adding a new replica.

Turns out that if you add a new replica to solrcloud:
1) Searchers are opened at least twice and possibly a third time
2) the first time is because of a new core coming online and opening searcher on an empty index – only firstSearcher event listeners are fired here
3) second time is after replication is complete and we have new index files available – firstSearcher event listeners are fired again because the old searcher opened on core load has already been closed and disposed so this is technically again a first searcher
4) third time happens after documents buffered during recovery are replayed – if there was no indexing happening on leader then this step is skipped – a newSearcher event is fired here because we had already opened a searcher in the last step

Now if instead of a new replica, a solr node is restarted then there can be upto four searcher opens – the additional open is because of log replay on startup.

So Solr spends a lot of time on unnecessary warming/autowarming on searchers that are discarded. It is not just warming because sometimes plugins such as SpellCheckComponent and SuggestComponent can also tie in to these listener events.

We can probably cut a few of them or at least defer the decision of whether to fire these listeners to places such as RecoveryStrategy which have a better idea of whether it is worth it.