Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8562

Speed up merging segments of points with data dimensions

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 7.7, 8.0
    • Fix Version/s: 7.7, 8.0
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Currently when merging segments of points with data dimensions, all dimensions are sorted and carried over down the tree even though only indexing dimensions are needed to build the BKD tree. This is needed so leaf node data can be compressed by common prefix.

      But when using MutablePointValues, this ordering is done at the leaf level so we can se a similar approach from data dimensions and delay the sorting at leaf level. This seems to speed up indexing time as well as reduce the storage needed for building the index.

       

       

       

        Attachments

        1. LUCENE-8562.patch
          10 kB
          Ignacio Vera
        2. LUCENE-8562.patch
          8 kB
          Ignacio Vera
        3. LUCENE-8562.patch
          8 kB
          Ignacio Vera
        4. LUCENE-8562.patch
          9 kB
          Ignacio Vera
        5. LUCENE-8562.patch
          11 kB
          Ignacio Vera
        6. LUCENE-8562.patch
          11 kB
          Ignacio Vera
        7. LUCENE-8562.patch
          10 kB
          Ignacio Vera

          Activity

            People

            • Assignee:
              ivera Ignacio Vera
              Reporter:
              ivera Ignacio Vera
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: