Lucene - Core
  1. Lucene - Core
  2. LUCENE-2138

Allow custom index readers when using IndexWriter.getReader

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.0
    • Fix Version/s: 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      This is needed for backwards compatible support with Solr, and is a spin-off from SOLR-1606.

      1. LUCENE-2138.patch
        2 kB
        Jason Rutherglen

        Issue Links

          Activity

          Hide
          Jason Rutherglen added a comment -

          Sorry if this spam's things, however it's unlikely that I'll work on these.

          Show
          Jason Rutherglen added a comment - Sorry if this spam's things, however it's unlikely that I'll work on these.
          Hide
          Michael McCandless added a comment -

          I think LUCENE-2026 should be largely unaffected by flex indexing, except that it now accepts the Codecs instance and must thread it down.

          Flex indexing lives "under" IndexWriter, except when the segment writing needs to create the segment, and then that part of IndexWriter is a consumer of the flex API.

          What makes me nervous about just doing this patch as-is, is that it's a change to IW's public API. For such changes I think it's worth spending some more time, looking forward, to how best to expose that API. I don't mind making such changes internal to lucene ("progress not perfection"), but for changes to the public API, we should give it some thought.

          There's been a number of requests (many from you!0 for customizing how readers are created (what class is used), what components they contain, how to associate caches with them, being notified when merges complete, being able to make deletions against them directly, controlling warming, etc. I'd like to put all these on the table and understand how we can make a coherent change, and pulling out ReaderPool (so that eg it's also shared with IndexReader.open) seems roughly like what we need to do to address these. LUCENE-2133 is also related here.

          Show
          Michael McCandless added a comment - I think LUCENE-2026 should be largely unaffected by flex indexing, except that it now accepts the Codecs instance and must thread it down. Flex indexing lives "under" IndexWriter, except when the segment writing needs to create the segment, and then that part of IndexWriter is a consumer of the flex API. What makes me nervous about just doing this patch as-is, is that it's a change to IW's public API. For such changes I think it's worth spending some more time, looking forward, to how best to expose that API. I don't mind making such changes internal to lucene ("progress not perfection"), but for changes to the public API, we should give it some thought. There's been a number of requests (many from you!0 for customizing how readers are created (what class is used), what components they contain, how to associate caches with them, being notified when merges complete, being able to make deletions against them directly, controlling warming, etc. I'd like to put all these on the table and understand how we can make a coherent change, and pulling out ReaderPool (so that eg it's also shared with IndexReader.open) seems roughly like what we need to do to address these. LUCENE-2133 is also related here.
          Hide
          Jason Rutherglen added a comment -

          I'm curious, will flex indexing affect development on
          LUCENE-2026? Do they overlap?

          What's the use case for 2026? I thought about how it could help
          with implementing LUCENE-1313, however these fairly large
          changes, sometimes consume more time then their worth? I think
          this patch, 2138 is simple enough to be included in 3.1 as is,
          then if there's an itch to be scratched by implementing 2026,
          2138 functionality is easy enough to add.

          Show
          Jason Rutherglen added a comment - I'm curious, will flex indexing affect development on LUCENE-2026 ? Do they overlap? What's the use case for 2026? I thought about how it could help with implementing LUCENE-1313 , however these fairly large changes, sometimes consume more time then their worth? I think this patch, 2138 is simple enough to be included in 3.1 as is, then if there's an itch to be scratched by implementing 2026, 2138 functionality is easy enough to add.
          Hide
          Michael McCandless added a comment -

          Could we maybe instead factor out ReaderPool from IW, and somehow enable this extensibility, there?

          This would be the first step in LUCENE-2026, I guess.

          The mergedSegmentWarmer should then also go into ReaderPool.

          Show
          Michael McCandless added a comment - Could we maybe instead factor out ReaderPool from IW, and somehow enable this extensibility, there? This would be the first step in LUCENE-2026 , I guess. The mergedSegmentWarmer should then also go into ReaderPool.
          Hide
          Jason Rutherglen added a comment -

          IndexWriter has an API to set and get the IndexReaderFactory which creates SegmentReaders.

          Show
          Jason Rutherglen added a comment - IndexWriter has an API to set and get the IndexReaderFactory which creates SegmentReaders.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jason Rutherglen
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development