[LUCENE-9030] Solr- and WordnetSynonymParser behaviour differs

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 8.2
• Fix Version/s: 9.0, 8.4
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Alan Woodward
• Reporter: Christoph Büscher
• Created: 28/Oct/19 19:29
• Updated: 28/Aug/22 15:52
• Resolved: 13/Nov/19 15:54

Description

Equivalent synonyms are showing up with different token types and ordering depending on whether the Solr format or the Wordnet format is used. A synonym set like

"woods, wood, forest" in Solr format leads to the following token stream (term and type) when analyzing the term "forest": 

"forest"/word, "woods"/SYNONYM, "wood" /SYNONYM

 

The following set in Wordnet format should give the same output (all terms are in the same synset), however all tokens are of type SYNONYM here and the original input token "forest" isn't the first one:

synonyms.txt:

s(100000001,1,'woods',n,1,0)
s(100000001,2,'wood',n,1,0)
s(100000001,3,'forest',n,1,0)

Token stream (term/type) when an

woods"/SYNONYM, "wood" /SYNONYM, "forest"/SYNONYM

I don't think this is intentional and is confusing (especially because the "original" input token type gets lost). I saw that the way the synsets are added to the SynonymMap in the respective parsers differes and have a PR that changes this.

Issue Links

• links to: GitHub Pull Request #981