Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.1, 7.0
    • Component/s: modules/spatial
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      LUCENE-7211 converted IntersectsPrefixTreeQuery to use DocIdSetBuilder, but didn't actually reduce garbage generation for my Solr index.

      Since something like 40% of my garbage (by space) is now attributed to DocIdSetBuilder.growBuffer, I charted a few different allocation strategies to see if I could tune things more.

      See here: http://i.imgur.com/7sXLAYv.jpg
      The jump-then-flatline at the right would be where DocIdSetBuilder gives up and allocates a FixedBitSet for a 100M-doc index. (The 1M-doc index curve/cutoff looked similar)

      Perhaps unsurprisingly, the 1/8th growth factor in ArrayUtil.oversize is terrible from an allocation standpoint if you're doing a lot of expansions, and is especially terrible when used to build a short-lived data structure like this one.
      By the time it goes with the FBS, it's allocated around twice as much memory for the buffer as it would have needed for just the FBS.

        Attachments

        1. allocation_plot.jpg
          39 kB
          Jeff Wartes
        2. LUCENE-7258.patch
          10 kB
          Adrien Grand
        3. LUCENE-7258-expanding.patch
          8 kB
          Adrien Grand
        4. LUCENE-7258-Tune-memory-allocation-rate-for-Intersec.patch
          6 kB
          Jeff Wartes
        5. LUCENE-7258-Tune-memory-allocation-rate-for-Intersec.patch
          7 kB
          Jeff Wartes

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jwartes Jeff Wartes
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: