Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5123

invert the codec postings API

    XMLWordPrintableJSON

Details

    • Wish
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 5.0, 6.0
    • None
    • None
    • New

    Description

      Currently FieldsConsumer/PostingsConsumer/etc is a "push" oriented api, e.g. FreqProxTermsWriter streams the postings at flush, and the default merge() takes the incoming codec api and filters out deleted docs and "pushes" via same api (but that can be overridden).

      It could be cleaner if we allowed for a "pull" model instead (like DocValues). For example, maybe FreqProxTermsWriter could expose a Terms of itself and just passed this to the codec consumer.

      This would give the codec more flexibility to e.g. do multiple passes if it wanted to do things like encode high-frequency terms more efficiently with a bitset-like encoding or other things...

      A codec can try to do things like this to some extent today, but its very difficult (look at buffering in Pulsing). We made this change with DV and it made a lot of interesting optimizations easy to implement...

      Attachments

        1. LUCENE-5123.patch
          60 kB
          Michael McCandless
        2. LUCENE-5123.patch
          95 kB
          Michael McCandless
        3. LUCENE-5123.patch
          113 kB
          Michael McCandless
        4. LUCENE-5123.patch
          202 kB
          Michael McCandless
        5. LUCENE-5123.patch
          233 kB
          Michael McCandless

        Issue Links

          Activity

            People

              mikemccand Michael McCandless
              rcmuir Robert Muir
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: