Lucene - Core
  1. Lucene - Core
  2. LUCENE-2402

Add an explicit method to invoke IndexDeletionPolicy

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      Today, if one uses an IDP which holds onto segments, such as SnapshotDeletionPolicy, or any other IDP in the tests, those segments are left in the index even if the IDP no longer references them, until IW.commit() is called (and actually does something). I'd like to add a specific method to IW which will invoke the IDP's logic and get rid of the unused segments w/o forcing the user to call IW.commit(). There are a couple of reasons for that:

      • Segments take up sometimes valuable HD space, and the application may wish to reclaim that space immediately. In some scenarios, the index is updated once in several hours (or even days), and waiting until then may not be acceptable.
      • I think it's a cleaner solution than waiting for the next commit() to happen. One can still wait for it if one wants, but otherwise it will give you the ability to immediately get rid of those segments.
      • TestSnapshotDeletionPolicy includes this code, which only strengthens (IMO) the need for such method:
        // Add one more document to force writer to commit a
        // final segment, so deletion policy has a chance to
        // delete again:
        Document doc = new Document();
        doc.add(new Field("content", "aaa", Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
        writer.addDocument(doc);
        

      If IW had an explicit method, that code would not need to exist there at all ...

      Here comes the fun part - naming the baby:

      • invokeDeletionPolicy – describes exactly what is going to happen. However, if the user did not set IDP at all (relying on default, which I think many do), users won't understand what is it.
      • deleteUnusedSegments - more user-friendly, assuming users understand what 'segments' are.

      BTW, IW already has deleteUnusedFiles() which only tries to delete unreferenced files that failed to delete before (such as on Windows, due to e.g. open readers). Perhaps instead of inventing a new name, we can change IW.deleteUnusedFiles to call IndexFileDeleter.checkpoint (instead of deletePendingFiles) which deletes those files + calls IDP.onCommit().

      1. LUCENE-2402.patch
        4 kB
        Shai Erera
      2. LUCENE-2402.patch
        5 kB
        Shai Erera

        Activity

        Grant Ingersoll made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12563486 ] jira [ 12584252 ]
        Mark Thomas made changes -
        Workflow jira [ 12508729 ] Default workflow, editable Closed status [ 12563486 ]
        Shai Erera made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Shai Erera made changes -
        Fix Version/s 3.1 [ 12314822 ]
        Shai Erera made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Shai Erera made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Shai Erera made changes -
        Attachment LUCENE-2402.patch [ 12442442 ]
        Shai Erera made changes -
        Field Original Value New Value
        Attachment LUCENE-2402.patch [ 12442400 ]
        Shai Erera created issue -

          People

          • Assignee:
            Shai Erera
            Reporter:
            Shai Erera
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development