[LUCENE-6638] Factor graph flattening out of SynonymFilter

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 6.0
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 29/Jun/15 21:17
• Updated: 28/Aug/22 14:37
• Resolved: 12/Jan/17 00:12

Description

Spinoff from LUCENE-6582.

SynonymFilter is very hairy, and has known nearly-impossible-to-fix bugs: it produces the wrong graph, both accepting too many phrases and not enough phrases, because it never creates new positions.

This makes improvements like LUCENE-6582, to fix some of its bugs, unnecessarily hard.

I'd like to pull out the graph flattening into its own token filter, and I think I have a starting patch that works. I started with the "sausagizer" stage on the branch from LUCENE-5012, but changed the approach so that it should not have so many adversarial cases.

I think this should make SynonymFilter quite a bit simpler, hopefully to the point where we can just fix its bugs already.

Issue Links

• is related to: LUCENE-6582 SynonymFilter should generate a correct (or, at least, better) graph