[LUCENE-6747] FingerprintFilter - a TokenFilter for clustering/linking purposes

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 5.4, 6.0
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Unassigned
• Reporter: Mark Harwood
• Created: 19/Aug/15 16:13
• Updated: 09/May/16 18:40
• Resolved: 27/Aug/15 16:06

Description

A TokenFilter that emits a single token which is a sorted, de-duplicated set of the input tokens.
This approach to normalizing text is used in tools like OpenRefine[1] and elsewhere [2] to help in clustering or linking texts.
The implementation proposed here has a an upper limit on the size of the combined token which is output.

[1] https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth
[2] https://rajmak.wordpress.com/2013/04/27/clustering-text-map-reduce-in-python/