[LUCENE-8395] WordDelimiterGraphFilter can incorrectly add holes to a TokenStream

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 7.5
• Lucene Fields: New
• Assignee: Alan Woodward
• Reporter: Alan Woodward
• Created: 12/Jul/18 11:35
• Updated: 28/Aug/22 15:32
• Resolved: 12/Jul/18 12:23

Description

If a token consists entirely of delimiter characters, then WordDelimiterGraphFilter will remove the token and insert a hole into the TokenStream.  However, it does this even if preserve_original is set, which results in an extra gap being added to the stream.