Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10499

Optimize SpatialHeatmapFacets DocSet to Bits conversion

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.6, 7.0
    • Component/s: spatial
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Using the facet.heatmap feature will internally constructs a Bits instance (to be passed down into Lucene) from the Solr-provided DocSet. The current procedure is sometimes fine but it can be optimized:

      • if docSet is equal to SolrIndexSearcher.getLiveDocs then use a null Bits, which signals to the Lucene code that it can skip reading the postings and simply use the doc freq when there are no deleted docs either. The difference is huge.
      • if docSet has size 0 then use Bits.MatchNoBits. After LUCENE-7787 is committed, this is a fast do-nothing optimization that could otherwise take a lot of time.
      • if the docSet is not a BitDocSet then build a FixedBitSet. Presently the code wraps a Bits around the DocSet, likely a SortedIntDocSet, which is a huge performance hit for large shards when this happens – log(N) over possibly thousands of doc IDs for each doc accumulation

      (This development was funded by the Harvard Center for Geographic Analysis as part of the HHypermap project)

        Activity

        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 991196216d2a0ba92ce8a772dd3a9a03eb29b241 in lucene-solr's branch refs/heads/master from David Smiley
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9911962 ]

        SOLR-10499: facet.heatmap DocSet to Bits optimizations

        Show
        jira-bot ASF subversion and git services added a comment - Commit 991196216d2a0ba92ce8a772dd3a9a03eb29b241 in lucene-solr's branch refs/heads/master from David Smiley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9911962 ] SOLR-10499 : facet.heatmap DocSet to Bits optimizations
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit eee1b389edac7aac73ab1e9e18b06ce77f7e1851 in lucene-solr's branch refs/heads/branch_6x from David Smiley
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=eee1b38 ]

        SOLR-10499: facet.heatmap DocSet to Bits optimizations

        (cherry picked from commit 9911962)

        Show
        jira-bot ASF subversion and git services added a comment - Commit eee1b389edac7aac73ab1e9e18b06ce77f7e1851 in lucene-solr's branch refs/heads/branch_6x from David Smiley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=eee1b38 ] SOLR-10499 : facet.heatmap DocSet to Bits optimizations (cherry picked from commit 9911962)

          People

          • Assignee:
            dsmiley David Smiley
            Reporter:
            dsmiley David Smiley
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development