[LUCENE-8332] New ConcatenateGraphFilter (move/rename CompletionTokenStream)

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 7.4
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 25/May/18 03:14
• Updated: 28/Aug/22 15:30
• Resolved: 05/Jun/18 03:33

Description

Lets move and rename the CompletionTokenStream in the suggest module into the analysis module renamed as ConcatenateGraphFilter. See comments in LUCENE-8323 leading to this idea. Such a TokenStream (or TokenFilter?) has several uses:

• for the suggest module
• by the SolrTextTagger for NER/ERD use cases – SOLR-12376
• for doing complete match search efficiently

It will need a factory – a TokenFilterFactory, even though we don't have a TokenFilter based subclass of TokenStream.

It appears there is no back-compat concern in it suddenly disappearing from the suggest module as it's marked experimental and it only seems to be public now perhaps due to some technicality (it has package level constructors).

Issue Links

• is duplicated by: LUCENE-8323 New ConcatenateFilter, a TokenFilter to concat/join tokens
• is required by: SOLR-12376 New TaggerRequestHandler (aka SolrTextTagger)
• links to: GitHub Pull Request #384