Lucene - Core
  1. Lucene - Core
  2. LUCENE-2955

Add utitily class to manage NRT reopening

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I created a simple class, NRTManager, that tries to abstract away some
      of the reopen logic when using NRT readers.

      You give it your IW, tell it min and max nanoseconds staleness you can
      tolerate, and it privately runs a reopen thread to periodically reopen
      the searcher.

      It subsumes the SearcherManager from LIA2. Besides running the reopen
      thread, it also adds the notion of a "generation" containing changes
      you've made. So eg it has addDocument, returning a long. You can
      then take that long value and pass it back to the getSearcher method
      and getSearcher will return a searcher that reflects the changes made
      in that generation.

      This gives your app the freedom to force "immediate" consistency (ie
      wait for the reopen) only for those searches that require it, like a
      verifier that adds a doc and then immediately searches for it, but
      also use "eventual consistency" for other searches.

      I want to also add support for the new "applyDeletions" option when
      pulling an NRT reader.

      Also, this is very new and I'm sure buggy – the concurrency is either
      wrong over overly-locking. But it's a start...

      1. LUCENE-2955.patch
        23 kB
        Michael McCandless
      2. LUCENE-2955.patch
        44 kB
        Michael McCandless
      3. LUCENE-2955.patch
        49 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch.

        The sources are in Lucene core now but I think we should commit into contrib somewhere since this is really a "wrapper" around core APIs.

        Show
        Michael McCandless added a comment - Patch. The sources are in Lucene core now but I think we should commit into contrib somewhere since this is really a "wrapper" around core APIs.
        Hide
        Robert Muir added a comment -

        bulk move 3.2 -> 3.3

        Show
        Robert Muir added a comment - bulk move 3.2 -> 3.3
        Hide
        Michael McCandless added a comment -

        New patch; I think it's ready to commit.

        You can now specify, when requesting the new searcher, whether deletes must have been applied.

        Show
        Michael McCandless added a comment - New patch; I think it's ready to commit. You can now specify, when requesting the new searcher, whether deletes must have been applied.
        Hide
        Simon Willnauer added a comment -

        Mike, nice work so far I have to admit that I really don't like the reopen thread. I think reopen in the background should be abstracted and the Reopen thread should not be part of the core manager. By default I think we should consult a ReopenStrategy on change and hijack indexing threads to reopen the reader. we can still sychronized the reopeing with a lock.tryLock() and by default go with a timed reopen policy. Thoughts?

        simon

        Show
        Simon Willnauer added a comment - Mike, nice work so far I have to admit that I really don't like the reopen thread. I think reopen in the background should be abstracted and the Reopen thread should not be part of the core manager. By default I think we should consult a ReopenStrategy on change and hijack indexing threads to reopen the reader. we can still sychronized the reopeing with a lock.tryLock() and by default go with a timed reopen policy. Thoughts? simon
        Hide
        Chris Male added a comment -

        I agree with Simon. I think providing a ReopenStrategy abstraction will be helpful.

        Show
        Chris Male added a comment - I agree with Simon. I think providing a ReopenStrategy abstraction will be helpful.
        Hide
        Jason Rutherglen added a comment -

        Perhaps we can merge this functionality with SOLR-2565 and/or SOLR-2566, such that Solr utilizes it for reader opening. However why would this issue use a background thread and Solr performs a max time reopen?

        Show
        Jason Rutherglen added a comment - Perhaps we can merge this functionality with SOLR-2565 and/or SOLR-2566 , such that Solr utilizes it for reader opening. However why would this issue use a background thread and Solr performs a max time reopen?
        Hide
        Michael McCandless added a comment -

        OK, new patch, folding in Simon's & Chris's feedback (thanks!).

        I pulled out the reopen thread into a separate class, so that one can
        now instantiate NRTManager but do their own reopening (no bg reopen
        thread). So eg if you want to hijack indexing threads to do reopen,
        you can.

        But if you want to simply reopen on a periodic basis with the bg
        thread, instantiate NRTManagerReopenThread, passing it the manager and
        your max and min staleness. Max staleness applies when no caller is
        waiting for a specific indexing change; min applies when one is.

        I didn't implement a ReopenStrategy... I think that should live
        "above" this class. But, I did add a WaitingListener so that such a
        reopener reopener can be notified when someone is waiting for a
        specific generation to be visible (NRTManagerReopenThread uses
        that).

        Show
        Michael McCandless added a comment - OK, new patch, folding in Simon's & Chris's feedback (thanks!). I pulled out the reopen thread into a separate class, so that one can now instantiate NRTManager but do their own reopening (no bg reopen thread). So eg if you want to hijack indexing threads to do reopen, you can. But if you want to simply reopen on a periodic basis with the bg thread, instantiate NRTManagerReopenThread, passing it the manager and your max and min staleness. Max staleness applies when no caller is waiting for a specific indexing change; min applies when one is. I didn't implement a ReopenStrategy... I think that should live "above" this class. But, I did add a WaitingListener so that such a reopener reopener can be notified when someone is waiting for a specific generation to be visible (NRTManagerReopenThread uses that).
        Hide
        Robert Muir added a comment -

        bulk close for 3.3

        Show
        Robert Muir added a comment - bulk close for 3.3

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development