Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-1458

Further steps towards flexible indexing


    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 4.0-ALPHA
    • Component/s: core/index
    • Labels:
    • Lucene Fields:


      I attached a very rough checkpoint of my current patch, to get early
      feedback. All tests pass, though back compat tests don't pass due to
      changes to package-private APIs plus certain bugs in tests that
      happened to work (eg call TermPostions.nextPosition() too many times,
      which the new API asserts against).

      [Aside: I think, when we commit changes to package-private APIs such
      that back-compat tests don't pass, we could go back, make a branch on
      the back-compat tag, commit changes to the tests to use the new
      package private APIs on that branch, then fix nightly build to use the
      tip of that branch?o]

      There's still plenty to do before this is committable! This is a
      rather large change:

      • Switches to a new more efficient terms dict format. This still
        uses tii/tis files, but the tii only stores term & long offset
        (not a TermInfo). At seek points, tis encodes term & freq/prox
        offsets absolutely instead of with deltas delta. Also, tis/tii
        are structured by field, so we don't have to record field number
        in every term.
        On first 1 M docs of Wikipedia, tii file is 36% smaller (0.99 MB
        -> 0.64 MB) and tis file is 9% smaller (75.5 MB -> 68.5 MB).
        RAM usage when loading terms dict index is significantly less
        since we only load an array of offsets and an array of String (no
        more TermInfo array). It should be faster to init too.
        This part is basically done.
      • Introduces modular reader codec that strongly decouples terms dict
        from docs/positions readers. EG there is no more TermInfo used
        when reading the new format.
        There's nice symmetry now between reading & writing in the codec
        chain – the current docs/prox format is captured in:
        FormatPostingsDocsWriter/Reader (.frq file) and
        FormatPostingsPositionsWriter/Reader (.prx file).

        This part is basically done.

      • Introduces a new "flex" API for iterating through the fields,
        terms, docs and positions:
        FieldProducer -> TermsEnum -> DocsEnum -> PostingsEnum

        This replaces TermEnum/Docs/Positions. SegmentReader emulates the
        old API on top of the new API to keep back-compat.

      Next steps:

      • Plug in new codecs (pulsing, pfor) to exercise the modularity /
        fix any hidden assumptions.
      • Expose new API out of IndexReader, deprecate old API but emulate
        old API on top of new one, switch all core/contrib users to the
        new API.
      • Maybe switch to AttributeSources as the base class for TermsEnum,
        DocsEnum, PostingsEnum – this would give readers API flexibility
        (not just index-file-format flexibility). EG if someone wanted
        to store payload at the term-doc level instead of
        term-doc-position level, you could just add a new attribute.
      • Test performance & iterate.


        1. LUCENE-1458-DocIdSetIterator.patch
          22 kB
          Uwe Schindler
        2. LUCENE-1458-DocIdSetIterator.patch
          21 kB
          Uwe Schindler
        3. LUCENE-1458-MTQ-BW.patch
          2 kB
          Uwe Schindler
        4. LUCENE-1458-NRQ.patch
          12 kB
          Uwe Schindler
        5. LUCENE-1458_rotate.patch
          4 kB
          Robert Muir
        6. LUCENE-1458_sortorder_bwcompat.patch
          3 kB
          Robert Muir
        7. LUCENE-1458_termenum_bwcompat.patch
          1 kB
          Robert Muir
        8. UnicodeTestCase.patch
          2 kB
          Robert Muir
        9. UnicodeTestCase.patch
          2 kB
          Robert Muir
        10. LUCENE-1458.patch
          883 kB
          Mark Miller
        11. LUCENE-1458.patch
          878 kB
          Mark Miller
        12. LUCENE-1458.patch
          909 kB
          Michael McCandless
        13. LUCENE-1458.patch
          895 kB
          Michael McCandless
        14. LUCENE-1458.patch
          886 kB
          Michael McCandless
        15. LUCENE-1458.patch
          1024 kB
          Mark Miller
        16. LUCENE-1458.patch
          1015 kB
          Mark Miller
        17. LUCENE-1458.tar.bz2
          1.93 MB
          Michael McCandless
        18. LUCENE-1458-back-compat.patch
          22 kB
          Michael McCandless
        19. LUCENE-1458.tar.bz2
          1.94 MB
          Michael McCandless
        20. LUCENE-1458-back-compat.patch
          22 kB
          Michael McCandless
        21. LUCENE-1458-back-compat.patch
          16 kB
          Michael McCandless
        22. LUCENE-1458.tar.bz2
          1.84 MB
          Michael McCandless
        23. LUCENE-1458-back-compat.patch
          16 kB
          Michael McCandless
        24. LUCENE-1458.tar.bz2
          1.83 MB
          Michael McCandless
        25. LUCENE-1458-back-compat.patch
          15 kB
          Michael McCandless
        26. LUCENE-1458.tar.bz2
          1.82 MB
          Michael McCandless
        27. LUCENE-1458.tar.bz2
          1.83 MB
          Michael McCandless
        28. LUCENE-1458-back-compat.patch
          15 kB
          Michael McCandless
        29. LUCENE-1458.tar.bz2
          1.80 MB
          Michael McCandless
        30. LUCENE-1458.patch
          360 kB
          Michael Busch
        31. LUCENE-1458.patch
          370 kB
          Michael McCandless
        32. LUCENE-1458.patch
          263 kB
          Michael McCandless
        33. LUCENE-1458.patch
          188 kB
          Michael McCandless
        34. LUCENE-1458.patch
          167 kB
          Michael McCandless
        35. LUCENE-1458.patch
          116 kB
          Michael McCandless

          Issue Links



              • Assignee:
                mikemccand Michael McCandless
                mikemccand Michael McCandless
              • Votes:
                1 Vote for this issue
                7 Start watching this issue


                • Created: