Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-3907

Improve the Edge/NGramTokenizer/Filters

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.4
    • None
    • New

    Description

      Our ngram tokenizers/filters could use some love. EG, they output ngrams in multiple passes, instead of "stacked", which messes up offsets/positions and requires too much buffering (can hit OOME for long tokens). They clip at 1024 chars (tokenizers) but don't (token filters). The split up surrogate pairs incorrectly.

      Attachments

        1. LUCENE-3907.patch
          43 kB
          Adrien Grand

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LUCENE-4641 Fix analyzer bugs documented in TestRandomChains

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LUCENE-3920 ngram tokenizer/filters create nonsense offsets if followed by a word combiner

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LUCENE-4810 Positions are incremented for each ngram in EdgeNGramTokenFilter

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SOLR-11894 [Ref-Guide] Removed parameter mentioned in EdgeNgram tokenizer doc

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jpountz Adrien Grand
              mikemccand Michael McCandless
              Votes:
              2 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: