[SOLR-12376] New TaggerRequestHandler (aka SolrTextTagger)

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 7.4
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 18/May/18 22:00
• Updated: 08/Jun/19 15:13
• Resolved: 05/Jun/18 18:31

Description

This issue introduces a new RequestHandler: TaggerRequestHandler, AKA the SolrTextTagger from the OpenSextant project https://github.com/OpenSextant/SolrTextTagger. It's used for named entity recognition (NER) of text past to it. It doesn't do any NLP (outside of Lucene text analysis) so it's said to be a "naive tagger", but it's definitely useful as-is and a more complete NER or ERD (entity recognition and disambiguation) system can be built with this as a key component. The SolrTextTagger has been used on queries for query-understanding, and it's been used on full-text, and it's been used on dictionaries that number tens of millions in size. Since it's small and has been used a bunch (including helping win an ERD competition and in Apache Stanbol), several people have asked me when or why isn't this in Solr yet. So here it is.

To use it, first you need a collection of documents that have a name-like field (short text) indexed with the ConcatenateFilter (LUCENE-8323) at the end. We call this the dictionary. Once that's in place, you simply post text to a TaggerRequestHandler and it returns the offset pairs into that text for matches in the dictionary along with the uniqueKey of the matching documents. It can also return other document data desired. That's the gist; I'll add more details on use to the Solr Reference Guide.

Issue Links

• requires: LUCENE-8323 New ConcatenateFilter, a TokenFilter to concat/join tokens
• requires: LUCENE-8332 New ConcatenateGraphFilter (move/rename CompletionTokenStream)