[LUCENE-9458] WordDelimiterGraphFilter (and non-graph) should tie-break order using end offset

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 8.7
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 11/Aug/20 21:32
• Updated: 28/Aug/22 16:05
• Resolved: 02/Oct/20 02:51

Description

WordDelimiterGraphFilter and WordDelimiterFilter do not consult the end offset in their sub-token ordering.  In the event of a tie-break, I propose the longer token come first.  This usually happens already, but not always, and so this also feels like an inconsistency when you see it.  This issue can be thought of as a bug fix to LUCENE-9006 or an improvement; I have no strong feelings on the issue classification.  Before reading further, definitely read that issue.

I see this is a problem when using CATENATE_ALL with either GENERATE_WORD_PARTS xor GENERATE_NUMBER_PARTS when the input ends with that part not being generated.  Consider the input: "other-9" and let's assume we want to catenate all, generate word parts, but nothing else (not numbers).  This should be tokenized in this order: "other9", "other" but today is emitted in reverse order.

 

Issue Links

• is related to: LUCENE-9006 Ensure WordDelimiterGraphFilter always emits catenateAll token early
• links to: GitHub Pull Request #1740