Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
4.10.4
-
OS: macOS Sierra (version 10.12.4)
Memory: 16GB
CPU: 2.9 GHz Intel Core i7
Java Version: 1.8
Description
The performance of sorting degrades significantly when the useFilterForSortedQuery is enabled, and there's no filter query specified.
Steps to Reproduce:
1. Set useFilterForSortedQuery=true in solrconfig.xml
2. Run a query to match and return a single document. Also add sorting
- Example /select?q=foo:123&sort=bar+desc
Having a large index (> 10 million documents), this yields to a slow response (a few hundreds of milliseconds on average) even when the resulting set consists of a single document.
Observation 1:
- Disabling useFilterForSortedQuery improves the performance to < 1ms
Observation 2:
- Removing the sort improves the performance to < 1ms
Observation 3:
- Keeping the sort, and adding any filter query (such as fq=*:*) improves the performance to < 1 ms.
After profiling SolrIndexSearcher.java found that the bottleneck is on
DocSet bigFilt = getDocSet(cmd.getFilterList());
when cmd.getFilterList()) is passed in as null. This is making getDocSet() function collect document ids every single time it is called without any caching.
1394 if (useFilterCache) { 1395 // now actually use the filter cache. 1396 // for large filters that match few documents, this may be 1397 // slower than simply re-executing the query. 1398 if (out.docSet == null) { 1399 out.docSet = getDocSet(cmd.getQuery(), cmd.getFilter()); 1400 DocSet bigFilt = getDocSet(cmd.getFilterList()); 1401 if (bigFilt != null) out.docSet = out.docSet.intersection(bigFilt); 1402 } 1403 // todo: there could be a sortDocSet that could take a list of 1404 // the filters instead of anding them first... 1405 // perhaps there should be a multi-docset-iterator 1406 sortDocSet(qr, cmd); 1407 }
Attachments
Attachments
Issue Links
- is related to
-
SOLR-12044 Optimize MatchAllDocsQuery for DocSets more
-
- Open
-