Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9330

Race condition between core reload and statistics request

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.5
    • Fix Version/s: 6.3, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Things happened that we execute this two requests consecutively in Solr 5.5:

      • Core reload: /admin/cores?action=RELOAD&core=coreName
      • Check core statistics: /coreName/admin/mbeans?stats=true

      And sometimes second request ends with this error:

      ERROR org.apache.solr.servlet.HttpSolrCall - null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
      	at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
      	at org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
      	at org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
      	at org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
      	at org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
      	at org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
      	at org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
      	at org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
      	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
      	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
      	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
      	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
      
      

      If my understanding of SolrCore internals is correct, it happens because of async nature of reload request:

      • New searcher is "registered" in separate thread
      • Old searcher is closed in same separate thread and only after new one is registered
      • When old searcher is closing, it removes itself from map with MBeans
      • If statistic requests happens before old searcher is completely removed from everywhere - exception can happen.

      What do you think if we will introduce new parameter for reload request which makes it fully synchronized? Basically it will force it to call

        SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] waitSearcher, boolean updateHandlerReopens) 

      with waitSearcher!= null

        Attachments

        1. SOLR-9390.patch
          9 kB
          Mikhail Khludnev
        2. too_sync.patch
          11 kB
          Andrey Kudryavtsev
        3. SOLR-9390.patch
          7 kB
          Andrey Kudryavtsev
        4. SOLR-9330.patch
          6 kB
          Mike Drob
        5. SOLR-9390.patch
          9 kB
          Mikhail Khludnev
        6. SOLR-9390.patch
          6 kB
          Mikhail Khludnev

          Issue Links

            Activity

              People

              • Assignee:
                mkhludnev Mikhail Khludnev
                Reporter:
                werder Andrey Kudryavtsev
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: