[SOLR-8559] FCS facet performance optimization

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 5.5, 6.0
• Fix Version/s: 5.5
• Component/s: faceting
• Labels: optimization performance
• Assignee: Dennis Gove
• Reporter: Keith Laban
• Created: 16/Jan/16 23:49
• Updated: 09/May/16 18:45
• Resolved: 22/Jan/16 21:43

Description

While profiling a large collection (multi-sharded billions of documents), I found that a fast (5-10ms query) which had no matches would take 20-30 seconds when doing facets even when facet.mincount=1

Profiling made it apparent that with facet.method=fcs 99% of the time was spent here.

queue.udpateTop gets called numOfSegments*numTerms, the worst case when every term is in every segment. This formula doesn't take into account whether or not any of the terms have a positive count with respect to the docset.

These optimizations are aimed to do two things:

1. When mincount>0 don't include segments which all terms have zero counts. This should significantly speed up processing when terms are high cardinality and the matching docset is small
2. FIXED TODO optimization: when mincount>0 move segment position the next non zero term value.

both of these changes will minimize the number of called needed to the slow updateTop call.