Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-5783

Can we stop opening a new searcher when the index hasn't changed?

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.8, 6.0
    • None
    • None

    Description

      I've been thinking recently about how/when we re-open searchers – and what the overhead of that is in terms of caches and what not – even if the underlying index hasn't changed.

      The particular real world case that got me thinking about this recently is when a deleteByQuery gets forwarded to all shards in a collection, and then the subsequent (soft)Commit (either auto or explicit) opens a new searcher – even if that shard was completley uneffected by the delete.

      It got me wondering: why don't re-use the same searcher when the index is unchanged?

      From what I can tell, we're basically 99% of the way there (in <nrtMode/>)...

      • IndexWriter.commit is already smart enough to short circut if there's nothing to commit
      • SolrCore.openNewSearcher already uses DirectoryReader.openIfChanged to see if the reader can be re-used.
      • for "realtime" purposes, SolrCore.openNewSearcher will return the existing searcher if it exists and the DirectoryReader hasn't changed

      ...The only reason I could think of for not always re-using the same searcher when the underlying DirectoryReader is identical (ie: that last bullet above) is in the situation where the "live" schema has changed – but that seems pretty trivial to account for.

      Is there any other reason why this wouldn't be a good idea for improving performance?

      Attachments

        1. SOLR-5783_harden_tests.patch
          5 kB
          Chris M. Hostetter
        2. SOLR-5783.patch
          16 kB
          Chris M. Hostetter
        3. SOLR-5783.patch
          14 kB
          Chris M. Hostetter
        4. SOLR-5783.patch
          7 kB
          Mark Miller
        5. SOLR-5783.patch
          3 kB
          Chris M. Hostetter

        Issue Links

          Activity

            People

              hossman Chris M. Hostetter
              hossman Chris M. Hostetter
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: