Lucene - Core
  1. Lucene - Core
  2. LUCENE-1737

Always use bulk-copy when merging stored fields and term vectors

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Lucene has nice optimizations in place during merging of stored fields
      (LUCENE-1043) and term vectors (LUCENE-1120) whereby the bytes are
      bulk copied to the new segmetn. This is much faster than decoding &
      rewriting one document at a time.

      However the optimization is rather brittle: it relies on the mapping
      of field name to number to be the same ("congruent") for the segment
      being merged.

      Unfortunately, the field mapping will be congruent only if the app
      adds the same fields in precisely the same order to each document.

      I think we should fix IndexWriter to assign the same field number for
      a given field that has been assigned in the past. Ie, when writing a
      new segment, we pre-seed the field numbers based on past segments.
      All other aspects of FieldInfo would remain fully dynamic.

      1. LUCENE-1737.patch
        16 kB
        Michael McCandless
      2. LUCENE-1737.patch
        4 kB
        Michael McCandless

        Issue Links

          Activity

          Michael McCandless created issue -
          Michael McCandless made changes -
          Field Original Value New Value
          Fix Version/s 2.9 [ 12312682 ]
          Michael McCandless made changes -
          Fix Version/s 4.0 [ 12314025 ]
          Michael McCandless made changes -
          Attachment LUCENE-1737.patch [ 12464788 ]
          Michael McCandless made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 3.1 [ 12314822 ]
          Resolution Fixed [ 1 ]
          Michael McCandless made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Michael McCandless made changes -
          Link This issue is blocked by LUCENE-2814 [ LUCENE-2814 ]
          Michael McCandless made changes -
          Attachment LUCENE-1737.patch [ 12466234 ]
          Michael McCandless made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Mark Thomas made changes -
          Workflow jira [ 12470110 ] Default workflow, editable Closed status [ 12563869 ]
          Mark Thomas made changes -
          Workflow Default workflow, editable Closed status [ 12563869 ] jira [ 12585380 ]
          Grant Ingersoll made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Michael McCandless
              Reporter:
              Michael McCandless
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development