Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2881

Track FieldInfo per segment instead of per-IW-session

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA, Realtime Branch, CSF branch
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Currently FieldInfo is tracked per IW session to guarantee consistent global field-naming / ordering. IW carries FI instances over from previous segments which also carries over field properties like isIndexed etc. While having consistent field ordering per IW session appears to be important due to bulk merging stored fields etc. carrying over other properties might become problematic with Lucene's Codec support. Codecs that rely on consistent properties in FI will fail if FI properties are carried over.

      The DocValuesCodec (DocValuesBranch) for instance writes files per segment and field (using the field id within the file name). Yet, if a segment has no DocValues indexed in a particular segment but a previous segment in the same IW session had DocValues, FieldInfo#docValues will be true since those values are reused from previous segments.

      We already work around this "limitation" in SegmentInfo with properties like hasVectors or hasProx which is really something we should manage per Codec & Segment. Ideally FieldInfo would be managed per Segment and Codec such that its properties are valid per segment. It also seems to be necessary to bind FieldInfoS to SegmentInfo logically since its really just per segment metadata.

        Attachments

        1. lucene-2881.patch
          85 kB
          Michael Busch
        2. lucene-2881.patch
          73 kB
          Michael Busch
        3. lucene-2881.patch
          73 kB
          Michael Busch
        4. lucene-2881.patch
          60 kB
          Michael Busch
        5. lucene-2881.patch
          47 kB
          Michael Busch
        6. LUCENE-2881.patch
          133 kB
          Simon Willnauer
        7. LUCENE-2881.patch
          123 kB
          Simon Willnauer
        8. LUCENE-2881.patch
          112 kB
          Simon Willnauer
        9. LUCENE-2881.patch
          91 kB
          Simon Willnauer

          Issue Links

            Activity

              People

              • Assignee:
                simonw Simon Willnauer
                Reporter:
                simonw Simon Willnauer
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: