Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-11769

Sorting performance degrades when useFilterForSortedQuery is enabled and there is no filter query specified



    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.10.4
    • 7.3
    • search
    • OS: macOS Sierra (version 10.12.4)
      Memory: 16GB
      CPU: 2.9 GHz Intel Core i7
      Java Version: 1.8


      The performance of sorting degrades significantly when the useFilterForSortedQuery is enabled, and there's no filter query specified.

      Steps to Reproduce:

      1. Set useFilterForSortedQuery=true in solrconfig.xml
      2. Run a query to match and return a single document. Also add sorting

      • Example /select?q=foo:123&sort=bar+desc

      Having a large index (> 10 million documents), this yields to a slow response (a few hundreds of milliseconds on average) even when the resulting set consists of a single document.

      Observation 1:

      • Disabling useFilterForSortedQuery improves the performance to < 1ms

      Observation 2:

      • Removing the sort improves the performance to < 1ms

      Observation 3:

      • Keeping the sort, and adding any filter query (such as fq=*:*) improves the performance to < 1 ms.

      After profiling SolrIndexSearcher.java found that the bottleneck is on
      DocSet bigFilt = getDocSet(cmd.getFilterList());

      when cmd.getFilterList()) is passed in as null. This is making getDocSet() function collect document ids every single time it is called without any caching.

      1394     if (useFilterCache) {
      1395       // now actually use the filter cache.
      1396       // for large filters that match few documents, this may be
      1397       // slower than simply re-executing the query.
      1398       if (out.docSet == null) {
      1399         out.docSet = getDocSet(cmd.getQuery(), cmd.getFilter());
      1400         DocSet bigFilt = getDocSet(cmd.getFilterList());
      1401         if (bigFilt != null) out.docSet = out.docSet.intersection(bigFilt);
      1402       }
      1403       // todo: there could be a sortDocSet that could take a list of
      1404       // the filters instead of anding them first...
      1405       // perhaps there should be a multi-docset-iterator
      1406       sortDocSet(qr, cmd);
      1407     }


        Issue Links



              dsmiley David Smiley
              bdeva Betim Deva
              0 Vote for this issue
              4 Start watching this issue