Lucene - Core
  1. Lucene - Core
  2. LUCENE-2158

NRT can temporarily lose deletions at high indexing rates

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.2, 3.0.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      OK, I found a sneaky case where NRT can temporarily lose deletions.
      The deletions aren't permanently lost - they are seen on the next
      opened NRT reader.

      It happens like this (in IW.getReader):

      • First flush() is called, to flush added docs & materialize the
        deletes.
      • The very next statement enters a sync'd block to open the reader,
        but, if indexing rate is very high, and threads get scheduled
        "appropriately", a "natural" flush (due to RAM buffer being full
        or flush doc count being reached) could be hit before the sync
        block is entered, in which case that 2nd flush won't materialize
        the deletes associated with it, and the returned NRT reader will
        only see its adds until it's next reopened.

      The fix is simple – we should materialize deletes inside the sync
      block, not during the flush.

      1. LUCENE-2158.patch
        3 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Attached patch.

        Jake/John, hopefully this explains what you guys were seeing in stress testing Lucene NRT!

        Show
        Michael McCandless added a comment - Attached patch. Jake/John, hopefully this explains what you guys were seeing in stress testing Lucene NRT!

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development