Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8310

Relax IWs check on pending deletes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 7.4, 8.0
    • Fix Version/s: 7.4, 8.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I recently fixed the check in IW to fail if there are pending deletes. After upgrading to a snapshot I realized the consequences of this check. It made most of our usage of IW to for instance prepare commit metadata, rollback to safe commit-points etc. impossible since we have to now busy wait on top of directory util all deletes are actually gone even though that we can guarantee that our history always goes forward. ie we are truly append-only in the sense of never reusing segment generations. The fix that I made was basically return false from a checkPendingDeletions in a directory wrapper to work around it.
      I do expect this to happen to a lot of lucene users even if they use IW correctly. My proposal is to make the check in IW a bit more sophisticated and only fail if there are pending deletes that are in the future from a generation perspective. We really don't care about files from the past. My patch checks the segment generation of each pending file which is safe since that is the same procedure we apply in IndexFileDeleter to inc reference etc. and only fail if the pending delete is in the future.

        Attachments

        1. LUCENE-8310.patch
          11 kB
          Simon Willnauer
        2. LUCENE-8310.patch
          19 kB
          Simon Willnauer
        3. LUCENE-8310.patch
          21 kB
          Simon Willnauer
        4. LUCENE-8310.patch
          21 kB
          Simon Willnauer

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                simonw Simon Willnauer
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: