Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-1073

Add unit test showing how to do a "live backup" of an index

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2
    • Fix Version/s: 2.3
    • Component/s: modules/examples
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      The question of how to backup an index comes up every so often on the
      lists. Backing up and index is also clearly an important fundamental
      admin task that many applications need to do for fault tolerance.

      In the past you were forced to stop & block all changes to your index,
      perform the backup, and then resume changes. But many applications
      cannot afford a potentially long pause in their indexing.

      With the addition of DeletionPolicy (LUCENE-710), it's now possible to
      do a "live backup", which means backup your index in the background
      without pausing ongoing changes to the index. This
      SnapshotDeletionPolicy just has to mark the chosen commit point as not
      deletable, until the backup finishes.

      1. LUCENE-1073.patch
        13 kB
        Michael McCandless
      2. LUCENE-1073.take2.patch
        13 kB
        Michael McCandless
      3. LUCENE-1073.take3.patch
        15 kB
        Michael McCandless

        Activity

        Hide
        mikemccand Michael McCandless added a comment -

        Attached patch that has a class (SnapshotDeletionPolicy) plus a unit
        test (TestSnapshotDeletionPolicy) showing how to use it to do a live
        backup.

        I also added a new public method "getFileNames()" to IndexCommitPoint
        (getting the filenames would otherwise requires package private access
        to SegmentInfo/s).

        All tests pass. I plan to commit in a day or two.

        Show
        mikemccand Michael McCandless added a comment - Attached patch that has a class (SnapshotDeletionPolicy) plus a unit test (TestSnapshotDeletionPolicy) showing how to use it to do a live backup. I also added a new public method "getFileNames()" to IndexCommitPoint (getting the filenames would otherwise requires package private access to SegmentInfo/s). All tests pass. I plan to commit in a day or two.
        Hide
        mikemccand Michael McCandless added a comment -

        Woops, the new testcase is failing on windows ... I'll track it down.

        Show
        mikemccand Michael McCandless added a comment - Woops, the new testcase is failing on windows ... I'll track it down.
        Hide
        mikemccand Michael McCandless added a comment -

        Attached new patch to fix the intermittant failure.

        Show
        mikemccand Michael McCandless added a comment - Attached new patch to fix the intermittant failure.
        Hide
        michaelbusch Michael Busch added a comment -

        Looks great, Mike!

        Maybe we should add SnapshotDeletionPolicy to the core (now it's an
        inner class in the test case). I bet people would like to use it when
        they implement backup functionalities in their apps.

        -Michael

        Show
        michaelbusch Michael Busch added a comment - Looks great, Mike! Maybe we should add SnapshotDeletionPolicy to the core (now it's an inner class in the test case). I bet people would like to use it when they implement backup functionalities in their apps. -Michael
        Hide
        mikemccand Michael McCandless added a comment -

        Thanks for reviewing Michael!

        Maybe we should add SnapshotDeletionPolicy to the core (now it's an
        inner class in the test case). I bet people would like to use it when
        they implement backup functionalities in their apps.

        OK I think that makes sense.

        I also found & fixed some thread safety issues with the test and added
        a missing synchronized in DocumentsWriter that the test uncovered.

        Attached new patch with these changes. I plan to commit in a day or
        two.

        Show
        mikemccand Michael McCandless added a comment - Thanks for reviewing Michael! Maybe we should add SnapshotDeletionPolicy to the core (now it's an inner class in the test case). I bet people would like to use it when they implement backup functionalities in their apps. OK I think that makes sense. I also found & fixed some thread safety issues with the test and added a missing synchronized in DocumentsWriter that the test uncovered. Attached new patch with these changes. I plan to commit in a day or two.
        Hide
        mikemccand Michael McCandless added a comment -

        I just committed this!

        Show
        mikemccand Michael McCandless added a comment - I just committed this!

          People

          • Assignee:
            mikemccand Michael McCandless
            Reporter:
            mikemccand Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development