Lucene - Core
  1. Lucene - Core
  2. LUCENE-3340

Buffered deletes are not flushed by RAM or count

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      When a segment is flushed, we will generally NOT flush the deletes, ie we simply buffer up the pending delete terms/queries, and the only apply them if 1) a segment is going to be merged (so we can remove the del docs in that segment), or 2) the buffered deletes' RAM exceeds 1/2 of IW's RAM limit when we are flushing a segment, or 3) the buffered deletes count exceeds IWC's maxBufferedDeleteTerms.

      But the latter 2 triggers are currently broken on trunk; I suspect (but I'm not sure) when we landed DWPT we introduced this bug.

      1. LUCENE-3340.patch
        11 kB
        Michael McCandless
      2. LUCENE-3340.patch
        6 kB
        Simon Willnauer
      3. LUCENE-3340.patch
        4 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch w/ 2 test cases showing the bug... they fail on trunk and pass on 3.x.

        Show
        Michael McCandless added a comment - Patch w/ 2 test cases showing the bug... they fail on trunk and pass on 3.x.
        Hide
        Simon Willnauer added a comment -

        here is a patch. The FlushPolicy only consulted the current global deletes buffer but didn't check the already frozen del packages. We also only checked if we need to apply the deletes during deleteQuery / deleteTerm and not during update.

        Show
        Simon Willnauer added a comment - here is a patch. The FlushPolicy only consulted the current global deletes buffer but didn't check the already frozen del packages. We also only checked if we need to apply the deletes during deleteQuery / deleteTerm and not during update.
        Hide
        Michael McCandless added a comment -

        Thanks Simon!

        I found another case where the deletes weren't flushing at the right time, and added a new test case (passes on 3.x and fails on trunk but passes w/ this patch).

        I think it's ready to commit!

        Show
        Michael McCandless added a comment - Thanks Simon! I found another case where the deletes weren't flushing at the right time, and added a new test case (passes on 3.x and fails on trunk but passes w/ this patch). I think it's ready to commit!
        Hide
        Simon Willnauer added a comment -

        I think it's ready to commit!

        +1 thanks for all the new testcases!

        Show
        Simon Willnauer added a comment - I think it's ready to commit! +1 thanks for all the new testcases!
        Hide
        Michael McCandless added a comment -

        Thanks Simon!

        Show
        Michael McCandless added a comment - Thanks Simon!

          People

          • Assignee:
            Unassigned
            Reporter:
            Michael McCandless
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development