[LUCENE-9006] Ensure WordDelimiterGraphFilter always emits catenateAll token early

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.4
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 15/Oct/19 14:19
• Updated: 28/Aug/22 15:51
• Resolved: 23/Oct/19 04:16

Description

Ideally, the first token of WDGF is the preserveOriginal (if configured to emit), and the second should be the catenateAll (if configured to emit). The deprecated WDF does this but WDGF can sometimes put the first other token earlier when there is a non-emitted candidate sub-token.

Example input "8-other" when only generateWordParts and catenateAll – not generateNumberParts. WDGF internally sees the '8' but moves on. Ultimately, the "other" token and the catenated "8other" will appear at the same internal position, which by luck fools the sorter to emit "other" first.

Issue Links

• is related to: LUCENE-8730 Ensure WordDelimiterGraphFilter always emits its original token first
• relates to: LUCENE-9458 WordDelimiterGraphFilter (and non-graph) should tie-break order using end offset
• links to: GitHub Pull Request #953