Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-3340

Buffered deletes are not flushed by RAM or count

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 4.0-ALPHA
        • Fix Version/s: 4.0-ALPHA
        • Component/s: core/index
        • Labels:
          None
        • Lucene Fields:
          New

          Description

          When a segment is flushed, we will generally NOT flush the deletes, ie we simply buffer up the pending delete terms/queries, and the only apply them if 1) a segment is going to be merged (so we can remove the del docs in that segment), or 2) the buffered deletes' RAM exceeds 1/2 of IW's RAM limit when we are flushing a segment, or 3) the buffered deletes count exceeds IWC's maxBufferedDeleteTerms.

          But the latter 2 triggers are currently broken on trunk; I suspect (but I'm not sure) when we landed DWPT we introduced this bug.

          1. Text File
            LUCENE-3340.patch
            11 kB
            Michael McCandless
          2. Text File
            LUCENE-3340.patch
            6 kB
            Simon Willnauer
          3. Text File
            LUCENE-3340.patch
            4 kB
            Michael McCandless

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            mikemccand Michael McCandless added a comment -

            Patch w/ 2 test cases showing the bug... they fail on trunk and pass on 3.x.

            Show
            mikemccand Michael McCandless added a comment - Patch w/ 2 test cases showing the bug... they fail on trunk and pass on 3.x.
            Hide
            Permalink
            simonw Simon Willnauer added a comment -

            here is a patch. The FlushPolicy only consulted the current global deletes buffer but didn't check the already frozen del packages. We also only checked if we need to apply the deletes during deleteQuery / deleteTerm and not during update.

            Show
            simonw Simon Willnauer added a comment - here is a patch. The FlushPolicy only consulted the current global deletes buffer but didn't check the already frozen del packages. We also only checked if we need to apply the deletes during deleteQuery / deleteTerm and not during update.
            Hide
            Permalink
            mikemccand Michael McCandless added a comment -

            Thanks Simon!

            I found another case where the deletes weren't flushing at the right time, and added a new test case (passes on 3.x and fails on trunk but passes w/ this patch).

            I think it's ready to commit!

            Show
            mikemccand Michael McCandless added a comment - Thanks Simon! I found another case where the deletes weren't flushing at the right time, and added a new test case (passes on 3.x and fails on trunk but passes w/ this patch). I think it's ready to commit!
            Hide
            Permalink
            simonw Simon Willnauer added a comment -

            I think it's ready to commit!

            +1 thanks for all the new testcases!

            Show
            simonw Simon Willnauer added a comment - I think it's ready to commit! +1 thanks for all the new testcases!
            Hide
            Permalink
            mikemccand Michael McCandless added a comment -

            Thanks Simon!

            Show
            mikemccand Michael McCandless added a comment - Thanks Simon!

              People

              • Assignee:
                Unassigned
                Reporter:
                mikemccand Michael McCandless
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development