Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-586

Very inefficient implementation of MultiTermDocs.skipTo

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1
    • Component/s: core/index, core/search
    • Labels:
      None

      Description

      In our application anytime the index was unoptimized/contained more than one segment there was a sharp drop in performance, which amounted to over 50ms per search on average. We would consistently see this drop anytime an index went from an optimized state to an unoptimized state.

      I tracked down the issue to the implementation of MultiTermDocs.skipTo function (found in MultiReader.java). Optimized indexes do not use this class during search but unoptimized indexes do. The comment on this function even explicitly states 'As yet unoptimized implementation.' It was implemented just by calling 'next' over and over so even if it knew it could skip ahead hundreds of thousands of hits it would not.

      So I re-implemented the function very similar to how the MultiTermDocs.next function was implemented and tested it out on or application for correctness and performance and it passed all our tests and the performance penalty of having multiple segments vanished. We have already put the new jar onto our production machines.

      Here is my implementation of skipTo, which closely mirrors the accepted implementation of 'next', please feel free to test it and commit it.

      /** Much more optimized implementation. Could be

      • optimized fairly easily to skip entire segments */
        public boolean skipTo(int target) throws IOException
        Unknown macro: { if (current != null && current.skipTo(target-base)) { return true; } else if (pointer < readers.length) { base = starts[pointer]; current = termDocs(pointer++); return skipTo(target); } else return false; }

        Activity

        Hide
        mikemccand Michael McCandless added a comment -

        Closing all issues that were resolved for 2.1.

        Show
        mikemccand Michael McCandless added a comment - Closing all issues that were resolved for 2.1.
        Hide
        yseeley@gmail.com Yonik Seeley added a comment -

        Thanks Andrew, I just committed this.

        Show
        yseeley@gmail.com Yonik Seeley added a comment - Thanks Andrew, I just committed this.

          People

          • Assignee:
            yseeley@gmail.com Yonik Seeley
            Reporter:
            ahudson Andrew Hudson
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development