Lucene - Core
  1. Lucene - Core
  2. LUCENE-2356

Enable setting the terms index divisor used by IndexWriter whenever it opens internal readers

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.3, 3.0.2, 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Opening a place holder issue... if all the refactoring being discussed don't make this possible, then we should add a setting to IWC to do so.

      Apps with very large numbers of unique terms must set the terms index divisor to control RAM usage.

      (NOTE: flex's RAM terms dict index RAM usage is more efficient, so this will help such apps).

      But, when IW resolves deletes internally it always uses default 1 terms index divisor, and the app cannot change that. Though one workaround is to call getReader(termInfosIndexDivisor) which will pool the reader with the right divisor.

      1. LUCENE-2356.patch
        6 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        I won't have any time to take this any time soon So if anyone has the itch, jump!

        Show
        Michael McCandless added a comment - I won't have any time to take this any time soon So if anyone has the itch, jump!
        Hide
        Michael McCandless added a comment -

        The above comment was on the wrong issue

        We should only do this issue if the ongoing ideas about refactoring IW/IR don't make controlling the terms index divisor possible, for readers opened by IW.

        Show
        Michael McCandless added a comment - The above comment was on the wrong issue We should only do this issue if the ongoing ideas about refactoring IW/IR don't make controlling the terms index divisor possible, for readers opened by IW.
        Hide
        Earwin Burrfoot added a comment -

        That's likely orthogonal.
        If you want all IW readers to have same divisor - shove it into IWC and it's all done.
        If you want to use different divisors when returning SR as a part of NRT reader and using it inside (say, for deletions) - okay, you'll have the ability to do that at the cost of partial SR reload - shove two settings into IWC and it's done.

        Show
        Earwin Burrfoot added a comment - That's likely orthogonal. If you want all IW readers to have same divisor - shove it into IWC and it's all done. If you want to use different divisors when returning SR as a part of NRT reader and using it inside (say, for deletions) - okay, you'll have the ability to do that at the cost of partial SR reload - shove two settings into IWC and it's done.
        Hide
        Michael McCandless added a comment -

        Simple patch – adds IWC.set/getReaderTermsIndexDivisor.

        Show
        Michael McCandless added a comment - Simple patch – adds IWC.set/getReaderTermsIndexDivisor.

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development