Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5252

add NGramSynonymTokenizer

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • modules/analysis
    • None
    • New

    Description

      I'd like to propose that we have another n-gram tokenizer which can process synonyms. That is NGramSynonymTokenizer. Note that in this ticket, the gram size is fixed, i.e. minGramSize = maxGramSize.

      Today, I think we have the following problems when using SynonymFilter with NGramTokenizer.
      For purpose of illustration, we have a synonym setting "ABC, DEFG" w/ expand=true and N = 2 (2-gram).

      1. There is no consensus (I think how we assign offsets to generated synonym tokens DE, EF and FG when expanding source token AB and BC.
      2. If the query pattern looks like ABCY, it cannot be matched even if there is a document "…ABCY…" in index when autoGeneratePhraseQueries set to true, because there is no "CY" token (but "GY" is there) in the index.

      NGramSynonymTokenizer can solve these problems by providing the following methods.

      • NGramSynonymTokenizer reads synonym settings (synonyms.txt) and it doesn't tokenize registered words. e.g.
      source text NGramTokenizer+SynonymFilter NGramSynonymTokenizer
      ABC AB/DE/BC/EF/FG ABC/DEFG
      • The back and forth of the registered words, NGramSynonymTokenizer generates extra tokens w/ posInc=0. e.g.
      source text NGramTokenizer+SynonymFilter NGramSynonymTokenizer
      XYZABC123 XY/YZ/ZA/AB/DE/BC/EF/C1/FG/12/23 XY/YZ/Z/ABC/DEFG/1/12/23

      In the above sample, "Z" and "1" are the extra tokens.

      Attachments

        1. LUCENE-5252_4x.patch
          84 kB
          Koji Sekiguchi
        2. LUCENE-5252_4x.patch
          84 kB
          Koji Sekiguchi
        3. LUCENE-5252_4x.patch
          84 kB
          Koji Sekiguchi
        4. LUCENE-5252_4x.patch
          83 kB
          Koji Sekiguchi
        5. LUCENE-5252_4x.patch
          20 kB
          Koji Sekiguchi

        Issue Links

          Activity

            People

              Unassigned Unassigned
              koji Koji Sekiguchi
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: