Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9149

Increase data dimension limit in BKD

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      LUCENE-8496 added selective indexing; the ability to designate the first K <= N dimensions for driving the construction of the BKD internal nodes. Follow on work stored the "data dimensions" for only the leaf nodes and only the "index dimensions" are stored for the internal nodes. While maxPointsInLeafNode is still important for managing the BKD heap memory footprint (thus we don't want this to get too large), I'd like to propose increasing the MAX_DIMENSIONS limit (to something not too crazy like 16; effectively doubling the index dimension limit) while maintaining the MAX_INDEX_DIMENSIONS at 8.

      Doing this will enable us to encode higher dimension data within a lower dimension index (e.g., 3D tessellated triangles as a 10 dimension point using only the first 6 dimensions for index construction)

       

        Attachments

        1. LUCENE-9149.patch
          138 kB
          Nick Knize

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                nknize Nick Knize
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m