Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-5894

Speed up high-cardinality facets with sparse counters

    XMLWordPrintableJSON

Details

    Description

      Multiple performance enhancements to Solr String faceting.

      • Sparse counters, switching the constant time overhead of extracting top-X terms with time overhead linear to result set size
      • Counter re-use for reduced garbage collection and lower per-call overhead
      • Optional counter packing, trading speed for space
      • Improved distribution count logic, greatly improving the performance of distributed faceting
      • In-segment threaded faceting
      • Regexp based white- and black-listing of facet terms
      • Heuristic faceting for large result sets

      Currently implemented for Solr 4.10. Source, detailed description and directly usable WAR at http://tokee.github.io/lucene-solr/

      This project has grown beyond a simple patch and will require a fair amount of co-operation with a committer to get into Solr. Splitting into smaller issues is a possibility.

      Attachments

        1. author_7M_tags_1852_logged_queries_warmed.png
          10 kB
          Toke Eskildsen
        2. SOLR-5894_test.zip
          53 kB
          Toke Eskildsen
        3. SOLR-5894_test.zip
          52 kB
          Toke Eskildsen
        4. SOLR-5894_test.zip
          48 kB
          Toke Eskildsen
        5. SOLR-5894_test.zip
          45 kB
          Toke Eskildsen
        6. SOLR-5894_test.zip
          45 kB
          Toke Eskildsen
        7. SOLR-5894.patch
          103 kB
          Toke Eskildsen
        8. SOLR-5894.patch
          102 kB
          Toke Eskildsen
        9. SOLR-5894.patch
          97 kB
          Toke Eskildsen
        10. SOLR-5894.patch
          108 kB
          Toke Eskildsen
        11. SOLR-5894.patch
          97 kB
          Toke Eskildsen
        12. SOLR-5894.patch
          75 kB
          Toke Eskildsen
        13. SOLR-5894.patch
          72 kB
          Toke Eskildsen
        14. SOLR-5894.patch
          17 kB
          Toke Eskildsen
        15. SOLR-5894.patch
          15 kB
          Toke Eskildsen
        16. sparse_2000000docs_fc_cutoff_20140403-145412.png
          11 kB
          Toke Eskildsen
        17. sparse_5000000docs_20140331-151918_multi.png
          13 kB
          Toke Eskildsen
        18. sparse_5000000docs_20140331-151918_single.png
          13 kB
          Toke Eskildsen
        19. sparse_50510000docs_20140328-152807.png
          13 kB
          Toke Eskildsen

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. SOLR-13807 Caching for term facet counts

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              toke Toke Eskildsen
              toke Toke Eskildsen
              Votes:
              14 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

                Created:
                Updated: