Description
Things happened that we execute this two requests consecutively in Solr 5.5:
- Core reload: /admin/cores?action=RELOAD&core=coreName
- Check core statistics: /coreName/admin/mbeans?stats=true
And sometimes second request ends with this error:
ERROR org.apache.solr.servlet.HttpSolrCall - null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) at org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331) at org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) at org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119) at org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404) at org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164) at org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134) at org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
If my understanding of SolrCore internals is correct, it happens because of async nature of reload request:
- New searcher is "registered" in separate thread
- Old searcher is closed in same separate thread and only after new one is registered
- When old searcher is closing, it removes itself from map with MBeans
- If statistic requests happens before old searcher is completely removed from everywhere - exception can happen.
What do you think if we will introduce new parameter for reload request which makes it fully synchronized? Basically it will force it to call
SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] waitSearcher, boolean updateHandlerReopens)
with waitSearcher!= null
Attachments
Attachments
Issue Links
- relates to
-
SOLR-4668 Request for core status during core reload causes 500 error due to IndexWriter AlreadyClosedException
- Resolved
-
SOLR-9699 CoreStatus requests can fail if executed during a core reload
- Resolved
-
SOLR-10489 StatsReloadRaceTest.testParallelReloadAndStats failures
- Resolved