Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-693

ConjunctionScorer - more tuneup

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1
    • None
    • core/search
    • None
    • Windows Server 2003 x64, Java 1.6, pretty large index

    • New

    Description

      (See also: #LUCENE-443)
      I did some profile testing with the new ConjuctionScorer in 2.1 and discovered a new bottleneck in ConjunctionScorer.sortScorers. The java.utils.Arrays.sort method is cloning the Scorers array on every sort, which is quite expensive on large indexes because of the size of the 'norms' array within, and isn't necessary.

      Here is one possible solution:

      private void sortScorers() {
      // squeeze the array down for the sort
      // if (length != scorers.length)

      { // Scorer[] temps = new Scorer[length]; // System.arraycopy(scorers, 0, temps, 0, length); // scorers = temps; // }

      insertionSort( scorers,length );
      // note that this comparator is not consistent with equals!
      // Arrays.sort(scorers, new Comparator() { // sort the array
      // public int compare(Object o1, Object o2)

      { // return ((Scorer)o1).doc() - ((Scorer)o2).doc(); // }

      // });

      first = 0;
      last = length - 1;
      }
      private void insertionSort( Scorer[] scores, int len)
      {
      for (int i=0; i<len; i++) {
      for (int j=i; j>0 && scores[j-1].doc() > scores[j].doc();j-- )

      { swap (scores, j, j-1); }

      }
      return;
      }
      private void swap(Object[] x, int a, int b)

      { Object t = x[a]; x[a] = x[b]; x[b] = t; }

      The squeezing of the array is no longer needed.
      We also initialized the Scorers array to 8 (instead of 2) to avoid having to grow the array for common queries, although this probably has less performance impact.

      This change added about 3% to query throughput in my testing.

      Peter

      Attachments

        1. conjunction.patch
          10 kB
          Yonik Seeley
        2. conjunction.patch
          15 kB
          Yonik Seeley
        3. conjunction.patch
          15 kB
          Yonik Seeley
        4. conjunction.patch.nosort1
          9 kB
          Yonik Seeley
        5. conjunction.patch
          16 kB
          Yonik Seeley

        Activity

          People

            Unassigned Unassigned
            pkeegan Peter Keegan
            Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment