Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9330

Race condition between core reload and statistics request

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 5.5
    • 6.3, 7.0
    • None
    • None

    Description

      Things happened that we execute this two requests consecutively in Solr 5.5:

      • Core reload: /admin/cores?action=RELOAD&core=coreName
      • Check core statistics: /coreName/admin/mbeans?stats=true

      And sometimes second request ends with this error:

      ERROR org.apache.solr.servlet.HttpSolrCall - null:org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
      	at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274)
      	at org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
      	at org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
      	at org.apache.lucene.index.FilterDirectoryReader.getVersion(FilterDirectoryReader.java:119)
      	at org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2404)
      	at org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:164)
      	at org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:134)
      	at org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:65)
      	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
      	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
      	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
      	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
      
      

      If my understanding of SolrCore internals is correct, it happens because of async nature of reload request:

      • New searcher is "registered" in separate thread
      • Old searcher is closed in same separate thread and only after new one is registered
      • When old searcher is closing, it removes itself from map with MBeans
      • If statistic requests happens before old searcher is completely removed from everywhere - exception can happen.

      What do you think if we will introduce new parameter for reload request which makes it fully synchronized? Basically it will force it to call

        SolrCore#getSearcher(boolean forceNew, boolean returnSearcher, final Future[] waitSearcher, boolean updateHandlerReopens) 

      with waitSearcher!= null

      Attachments

        1. SOLR-9330.patch
          6 kB
          Mike Drob
        2. SOLR-9390.patch
          9 kB
          Mikhail Khludnev
        3. SOLR-9390.patch
          7 kB
          Andrey Kudryavtsev
        4. SOLR-9390.patch
          9 kB
          Mikhail Khludnev
        5. SOLR-9390.patch
          6 kB
          Mikhail Khludnev
        6. too_sync.patch
          11 kB
          Andrey Kudryavtsev

        Issue Links

          Activity

            People

              mkhl Mikhail Khludnev
              werder Andrey Kudryavtsev
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: