[SOLR-9330] Race condition between core reload and statistics request

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.5
• Fix Version/s: 6.3, 7.0
• Assignee: Mikhail Khludnev
• Reporter: Andrey Kudryavtsev
• Created: 22/Jul/16 16:59
• Updated: 08/Oct/19 14:51
• Resolved: 23/Sep/16 08:53

Description

Things happened that we execute this two requests consecutively in Solr 5.5:

• Core reload: /admin/cores?action=RELOAD&core=coreName
• Check core statistics: /coreName/admin/mbeans?stats=true

And sometimes second request ends with this error:

ERROR org.apache.solr.servlet.HttpSolrCall - null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
	at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
	at org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
	at org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
	at org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
	at org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
	at org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
	at org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
	at org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)

If my understanding of SolrCore internals is correct, it happens because of async nature of reload request:

• New searcher is "registered" in separate thread
• Old searcher is closed in same separate thread and only after new one is registered
• When old searcher is closing, it removes itself from map with MBeans
• If statistic requests happens before old searcher is completely removed from everywhere - exception can happen.

What do you think if we will introduce new parameter for reload request which makes it fully synchronized? Basically it will force it to call

  SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] waitSearcher, boolean updateHandlerReopens) 

with waitSearcher!= null

Issue Links

• relates to: SOLR-4668 Request for core status during core reload causes 500 error due to IndexWriter AlreadyClosedException
• relates to: SOLR-9699 CoreStatus requests can fail if executed during a core reload
• relates to: SOLR-10489 StatsReloadRaceTest.testParallelReloadAndStats failures