Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-1007

Flexibility to turn on/off any flush triggers

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      See discussion at http://www.gossamer-threads.com/lists/lucene/java-dev/53186

      Provide the flexibility to turn on/off any flush triggers - ramBufferSize, maxBufferedDocs and maxBufferedDeleteTerms. One of ramBufferSize and maxBufferedDocs must be enabled.

      1. LUCENE-1007.take3.patch
        20 kB
        Michael McCandless
      2. LUCENE-1007.take2.patch
        19 kB
        Ning Li
      3. LUCENE-1007.patch
        14 kB
        Ning Li

        Activity

        Hide
        ningli Ning Li added a comment -

        Just got around to do the patch:

        • The patch includes changes to IndexWriter and DocumentsWriter to provide the flexibility to turn on/off any flush triggers.
        • Necessary changes to a couple of unit tests.
        • Also remove some unused imports.
        • All unit tests pass.

        One question: Should we count buffered delete terms towards ram buffer used? Feel like we should. On the other hand, numBytesUsed only counts ram space which can be recycled.

        Show
        ningli Ning Li added a comment - Just got around to do the patch: The patch includes changes to IndexWriter and DocumentsWriter to provide the flexibility to turn on/off any flush triggers. Necessary changes to a couple of unit tests. Also remove some unused imports. All unit tests pass. One question: Should we count buffered delete terms towards ram buffer used? Feel like we should. On the other hand, numBytesUsed only counts ram space which can be recycled.
        Hide
        mikemccand Michael McCandless added a comment -

        Thanks Ning! I will have a look at the patch.

        > One question: Should we count buffered delete terms towards ram
        > buffer used? Feel like we should. On the other hand, numBytesUsed
        > only counts ram space which can be recycled.

        I think we should still count this ram usage with numBytesUsed? Even
        though it's not directly recycled, this memory usage does still "last"
        for a very long time (ie longer than a single document's processing,
        until flush happens), so I think it should be counted.

        Show
        mikemccand Michael McCandless added a comment - Thanks Ning! I will have a look at the patch. > One question: Should we count buffered delete terms towards ram > buffer used? Feel like we should. On the other hand, numBytesUsed > only counts ram space which can be recycled. I think we should still count this ram usage with numBytesUsed? Even though it's not directly recycled, this memory usage does still "last" for a very long time (ie longer than a single document's processing, until flush happens), so I think it should be counted.
        Hide
        ningli Ning Li added a comment -

        Take2 counts buffered delete terms towards ram buffer used. A test case for it is added.

        Show
        ningli Ning Li added a comment - Take2 counts buffered delete terms towards ram buffer used. A test case for it is added.
        Hide
        mikemccand Michael McCandless added a comment -

        This patch looks good, thanks Ning! I made a few small changes,
        including how RAM usage of buffered delete term is calculated.
        Attached take3.

        I plan to commit in a day or two.

        Show
        mikemccand Michael McCandless added a comment - This patch looks good, thanks Ning! I made a few small changes, including how RAM usage of buffered delete term is calculated. Attached take3. I plan to commit in a day or two.
        Hide
        ningli Ning Li added a comment -

        One more thing about the approximation of actual bytes used for buffered delete term: just remember Integer.SIZE returns the number of bits used, should convert it to number of bytes.

        Show
        ningli Ning Li added a comment - One more thing about the approximation of actual bytes used for buffered delete term: just remember Integer.SIZE returns the number of bits used, should convert it to number of bytes.
        Hide
        mikemccand Michael McCandless added a comment -

        Whoa, good catch! OK I'll divide by 8.

        Show
        mikemccand Michael McCandless added a comment - Whoa, good catch! OK I'll divide by 8.
        Hide
        mikemccand Michael McCandless added a comment -

        I just committed this. Thanks Ning and Doron!

        Show
        mikemccand Michael McCandless added a comment - I just committed this. Thanks Ning and Doron!

          People

          • Assignee:
            mikemccand Michael McCandless
            Reporter:
            ningli Ning Li
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development