Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7639

Use Suffix Arrays for fast search with leading asterisks

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      If query term starts with asterisks FST checks all words in the dictionary so request processing speed falls down. This problem can be solved with Suffix Array approach. Luckily, Suffix Array can be constructed after Lucene start from existing index. Unfortunately, Suffix Arrays requires a lot of RAM so we can use it only when special flag is set:

      -Dsolr.suffixArray.enable=true

      It is possible to speed up Suffix Array initialization using several threads, so we can control number of threads with

      -Dsolr.suffixArray.initialization_treads_count=5

      This system property can be omitted, the default value is 5.

      Attached patch is the suggested implementation for SuffixArray support, it works for all terms starting with asterisks with at least 3 consequent non-wildcard characters. This patch do not change search results and affects only performance issues.

      Update
      suffix-arra-2.patch is an improved version of the first patch, system properties for it are following::

      lucene.suffixArray.enable - true, if you want to enable Suffix Array support. Default value - false.
      lucene.suffixArray.initializationThreadsCount - number of threads for Suffix Array initialization, if you set 0 - no additional threads used. Default value - 5.

        Attachments

        1. suffix-array.patch
          37 kB
          Yakov Sirotkin
        2. suffix-array-2.patch
          36 kB
          Yakov Sirotkin

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                yasha Yakov Sirotkin
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: