Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8395

WordDelimiterGraphFilter can incorrectly add holes to a TokenStream

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 7.5
    • None
    • None
    • New

    Description

      If a token consists entirely of delimiter characters, then WordDelimiterGraphFilter will remove the token and insert a hole into the TokenStream.  However, it does this even if preserve_original is set, which results in an extra gap being added to the stream.

      Attachments

        1. LUCENE-8395.patch
          3 kB
          Alan Woodward

        Activity

          People

            romseygeek Alan Woodward
            romseygeek Alan Woodward
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: