[LUCENE-5012] Make graph-based TokenFilters easier

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 21/May/13 20:03
• Updated: 28/Aug/22 13:46

Description

SynonymFilter has two limitations today:

• It cannot create positions, so eg dns -> domain name service
  creates blatantly wrong highlights (SOLR-3390, LUCENE-4499 and
  others).

• It cannot consume a graph, so e.g. if you try to apply synonyms
  after Kuromoji tokenizer I'm not sure what will happen.

I've thought about how to fix these issues but it's really quite
difficult with the current PosInc/PosLen graph representation, so I'd
like to explore an alternative approach.