Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2356

Enable setting the terms index divisor used by IndexWriter whenever it opens internal readers

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.9.3, 3.0.2, 3.1, 4.0-ALPHA
    • core/index
    • None
    • New

    Description

      Opening a place holder issue... if all the refactoring being discussed don't make this possible, then we should add a setting to IWC to do so.

      Apps with very large numbers of unique terms must set the terms index divisor to control RAM usage.

      (NOTE: flex's RAM terms dict index RAM usage is more efficient, so this will help such apps).

      But, when IW resolves deletes internally it always uses default 1 terms index divisor, and the app cannot change that. Though one workaround is to call getReader(termInfosIndexDivisor) which will pool the reader with the right divisor.

      Attachments

        1. LUCENE-2356.patch
          6 kB
          Michael McCandless

        Activity

          I won't have any time to take this any time soon So if anyone has the itch, jump!

          mikemccand Michael McCandless added a comment - I won't have any time to take this any time soon So if anyone has the itch, jump!

          The above comment was on the wrong issue

          We should only do this issue if the ongoing ideas about refactoring IW/IR don't make controlling the terms index divisor possible, for readers opened by IW.

          mikemccand Michael McCandless added a comment - The above comment was on the wrong issue We should only do this issue if the ongoing ideas about refactoring IW/IR don't make controlling the terms index divisor possible, for readers opened by IW.

          That's likely orthogonal.
          If you want all IW readers to have same divisor - shove it into IWC and it's all done.
          If you want to use different divisors when returning SR as a part of NRT reader and using it inside (say, for deletions) - okay, you'll have the ability to do that at the cost of partial SR reload - shove two settings into IWC and it's done.

          earwin Earwin Burrfoot added a comment - That's likely orthogonal. If you want all IW readers to have same divisor - shove it into IWC and it's all done. If you want to use different divisors when returning SR as a part of NRT reader and using it inside (say, for deletions) - okay, you'll have the ability to do that at the cost of partial SR reload - shove two settings into IWC and it's done.

          Simple patch – adds IWC.set/getReaderTermsIndexDivisor.

          mikemccand Michael McCandless added a comment - Simple patch – adds IWC.set/getReaderTermsIndexDivisor.
          tomoko Tomoko Uchida added a comment -

          This issue was moved to GitHub issue: #3432.

          tomoko Tomoko Uchida added a comment - This issue was moved to GitHub issue: #3432 .

          People

            mikemccand Michael McCandless
            mikemccand Michael McCandless
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: