Lucene - Core
  1. Lucene - Core
  2. LUCENE-2356

Enable setting the terms index divisor used by IndexWriter whenever it opens internal readers

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.3, 3.0.2, 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Opening a place holder issue... if all the refactoring being discussed don't make this possible, then we should add a setting to IWC to do so.

      Apps with very large numbers of unique terms must set the terms index divisor to control RAM usage.

      (NOTE: flex's RAM terms dict index RAM usage is more efficient, so this will help such apps).

      But, when IW resolves deletes internally it always uses default 1 terms index divisor, and the app cannot change that. Though one workaround is to call getReader(termInfosIndexDivisor) which will pool the reader with the right divisor.

      1. LUCENE-2356.patch
        6 kB
        Michael McCandless

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        64d 23h 47m 1 Michael McCandless 02/Jun/10 11:55
        Resolved Resolved Closed Closed
        15d 21h 8m 1 Uwe Schindler 18/Jun/10 09:04
        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12564292 ] jira [ 12584104 ]
        Mark Thomas made changes -
        Workflow jira [ 12503241 ] Default workflow, editable Closed status [ 12564292 ]
        Uwe Schindler made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Michael McCandless made changes -
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Michael McCandless made changes -
        Assignee Michael McCandless [ mikemccand ]
        Michael McCandless made changes -
        Attachment LUCENE-2356.patch [ 12446064 ]
        Hide
        Michael McCandless added a comment -

        Simple patch – adds IWC.set/getReaderTermsIndexDivisor.

        Show
        Michael McCandless added a comment - Simple patch – adds IWC.set/getReaderTermsIndexDivisor.
        Michael McCandless made changes -
        Fix Version/s 2.9.3 [ 12314799 ]
        Fix Version/s 3.0.2 [ 12314798 ]
        Fix Version/s 3.1 [ 12314822 ]
        Uwe Schindler made changes -
        Fix Version/s 4.0 [ 12314025 ]
        Fix Version/s 3.1 [ 12314822 ]
        Robert Muir made changes -
        Component/s Index [ 12310232 ]
        Uwe Schindler made changes -
        Field Original Value New Value
        Fix Version/s 4.0.0 [ 12314822 ]
        Fix Version/s 3.1 [ 12314025 ]
        Hide
        Earwin Burrfoot added a comment -

        That's likely orthogonal.
        If you want all IW readers to have same divisor - shove it into IWC and it's all done.
        If you want to use different divisors when returning SR as a part of NRT reader and using it inside (say, for deletions) - okay, you'll have the ability to do that at the cost of partial SR reload - shove two settings into IWC and it's done.

        Show
        Earwin Burrfoot added a comment - That's likely orthogonal. If you want all IW readers to have same divisor - shove it into IWC and it's all done. If you want to use different divisors when returning SR as a part of NRT reader and using it inside (say, for deletions) - okay, you'll have the ability to do that at the cost of partial SR reload - shove two settings into IWC and it's done.
        Hide
        Michael McCandless added a comment -

        The above comment was on the wrong issue

        We should only do this issue if the ongoing ideas about refactoring IW/IR don't make controlling the terms index divisor possible, for readers opened by IW.

        Show
        Michael McCandless added a comment - The above comment was on the wrong issue We should only do this issue if the ongoing ideas about refactoring IW/IR don't make controlling the terms index divisor possible, for readers opened by IW.
        Hide
        Michael McCandless added a comment -

        I won't have any time to take this any time soon So if anyone has the itch, jump!

        Show
        Michael McCandless added a comment - I won't have any time to take this any time soon So if anyone has the itch, jump!
        Michael McCandless created issue -

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development