Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-1458

Further steps towards flexible indexing

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I attached a very rough checkpoint of my current patch, to get early
      feedback. All tests pass, though back compat tests don't pass due to
      changes to package-private APIs plus certain bugs in tests that
      happened to work (eg call TermPostions.nextPosition() too many times,
      which the new API asserts against).

      [Aside: I think, when we commit changes to package-private APIs such
      that back-compat tests don't pass, we could go back, make a branch on
      the back-compat tag, commit changes to the tests to use the new
      package private APIs on that branch, then fix nightly build to use the
      tip of that branch?o]

      There's still plenty to do before this is committable! This is a
      rather large change:

      • Switches to a new more efficient terms dict format. This still
        uses tii/tis files, but the tii only stores term & long offset
        (not a TermInfo). At seek points, tis encodes term & freq/prox
        offsets absolutely instead of with deltas delta. Also, tis/tii
        are structured by field, so we don't have to record field number
        in every term.
        .
        On first 1 M docs of Wikipedia, tii file is 36% smaller (0.99 MB
        -> 0.64 MB) and tis file is 9% smaller (75.5 MB -> 68.5 MB).
        .
        RAM usage when loading terms dict index is significantly less
        since we only load an array of offsets and an array of String (no
        more TermInfo array). It should be faster to init too.
        .
        This part is basically done.
      • Introduces modular reader codec that strongly decouples terms dict
        from docs/positions readers. EG there is no more TermInfo used
        when reading the new format.
        .
        There's nice symmetry now between reading & writing in the codec
        chain – the current docs/prox format is captured in:
        FormatPostingsTermsDictWriter/Reader
        FormatPostingsDocsWriter/Reader (.frq file) and
        FormatPostingsPositionsWriter/Reader (.prx file).
        

        This part is basically done.

      • Introduces a new "flex" API for iterating through the fields,
        terms, docs and positions:
        FieldProducer -> TermsEnum -> DocsEnum -> PostingsEnum
        

        This replaces TermEnum/Docs/Positions. SegmentReader emulates the
        old API on top of the new API to keep back-compat.

      Next steps:

      • Plug in new codecs (pulsing, pfor) to exercise the modularity /
        fix any hidden assumptions.
      • Expose new API out of IndexReader, deprecate old API but emulate
        old API on top of new one, switch all core/contrib users to the
        new API.
      • Maybe switch to AttributeSources as the base class for TermsEnum,
        DocsEnum, PostingsEnum – this would give readers API flexibility
        (not just index-file-format flexibility). EG if someone wanted
        to store payload at the term-doc level instead of
        term-doc-position level, you could just add a new attribute.
      • Test performance & iterate.

        Attachments

        1. UnicodeTestCase.patch
          2 kB
          Robert Muir
        2. UnicodeTestCase.patch
          2 kB
          Robert Muir
        3. LUCENE-1458-NRQ.patch
          12 kB
          Uwe Schindler
        4. LUCENE-1458-MTQ-BW.patch
          2 kB
          Uwe Schindler
        5. LUCENE-1458-DocIdSetIterator.patch
          21 kB
          Uwe Schindler
        6. LUCENE-1458-DocIdSetIterator.patch
          22 kB
          Uwe Schindler
        7. LUCENE-1458-back-compat.patch
          15 kB
          Michael McCandless
        8. LUCENE-1458-back-compat.patch
          15 kB
          Michael McCandless
        9. LUCENE-1458-back-compat.patch
          16 kB
          Michael McCandless
        10. LUCENE-1458-back-compat.patch
          16 kB
          Michael McCandless
        11. LUCENE-1458-back-compat.patch
          22 kB
          Michael McCandless
        12. LUCENE-1458-back-compat.patch
          22 kB
          Michael McCandless
        13. LUCENE-1458.tar.bz2
          1.80 MB
          Michael McCandless
        14. LUCENE-1458.tar.bz2
          1.83 MB
          Michael McCandless
        15. LUCENE-1458.tar.bz2
          1.82 MB
          Michael McCandless
        16. LUCENE-1458.tar.bz2
          1.83 MB
          Michael McCandless
        17. LUCENE-1458.tar.bz2
          1.84 MB
          Michael McCandless
        18. LUCENE-1458.tar.bz2
          1.94 MB
          Michael McCandless
        19. LUCENE-1458.tar.bz2
          1.93 MB
          Michael McCandless
        20. LUCENE-1458.patch
          116 kB
          Michael McCandless
        21. LUCENE-1458.patch
          167 kB
          Michael McCandless
        22. LUCENE-1458.patch
          188 kB
          Michael McCandless
        23. LUCENE-1458.patch
          263 kB
          Michael McCandless
        24. LUCENE-1458.patch
          370 kB
          Michael McCandless
        25. LUCENE-1458.patch
          360 kB
          Michael Busch
        26. LUCENE-1458.patch
          1015 kB
          Mark Miller
        27. LUCENE-1458.patch
          1024 kB
          Mark Miller
        28. LUCENE-1458.patch
          886 kB
          Michael McCandless
        29. LUCENE-1458.patch
          895 kB
          Michael McCandless
        30. LUCENE-1458.patch
          909 kB
          Michael McCandless
        31. LUCENE-1458.patch
          878 kB
          Mark Miller
        32. LUCENE-1458.patch
          883 kB
          Mark Miller
        33. LUCENE-1458_termenum_bwcompat.patch
          1 kB
          Robert Muir
        34. LUCENE-1458_sortorder_bwcompat.patch
          3 kB
          Robert Muir
        35. LUCENE-1458_rotate.patch
          4 kB
          Robert Muir

          Issue Links

            Activity

              People

              • Assignee:
                mikemccand Michael McCandless
                Reporter:
                mikemccand Michael McCandless
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: