Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2770

Optimize SegmentMerger to work on atomic (Segment)Readers where possible

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      This is a spin-off from LUCENE-2769:

      Currently SegmentMerger has some optimizations when it merges segments that are SegmentReaders (e.g. when doing normal indexing or optimizing). But when you do IndexWriter.addIndexes(IndexReader...) the listed IndexReaders may not really be per-segment. SegmentMerger should track down all passed in reads down to the lowest level (Segment)Reader (or other atomic readers like SlowMultiReaderWrapper) and then merge. We can then remove most MultiFields usage (except term merging itsself) and clean up the code.

      This especially saves lots of memory for merging norms, as no longer the duplicate norms arrays are created when MultiReaders are used!

        Attachments

        1. LUCENE-2770-optimizeNormsMerging.patch
          2 kB
          Uwe Schindler
        2. LUCENE-2770-3x.patch
          3 kB
          Uwe Schindler
        3. LUCENE-2770-3x.patch
          7 kB
          Uwe Schindler
        4. LUCENE-2770.patch
          8 kB
          Uwe Schindler
        5. LUCENE-2770.patch
          9 kB
          Uwe Schindler
        6. LUCENE-2770.patch
          11 kB
          Uwe Schindler

          Issue Links

            Activity

              People

              • Assignee:
                thetaphi Uwe Schindler
                Reporter:
                thetaphi Uwe Schindler
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: