[LUCENE-5548] Improve flexibility and testability of the classification module

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: modules/classification
• Labels: gsoc2014 mentor
• Lucene Fields: New
• Assignee: Tommaso Teofili
• Reporter: Tommaso Teofili
• Created: 21/Mar/14 15:45
• Updated: 28/Aug/22 14:03
• Resolved: 13/Oct/21 16:35

Description

Lucene classification module's flexibility and capabilities may be improved with the following:

• make it possible to use them "online" (or provide an online version of them) so that if the underlying index(reader) is updated the classifier doesn't need to be trained again to take into account newly added docs
• eventually pass a different Analyzer together with the text to be classified (or directly a TokenStream) to specify custom tokenization/filtering.
• normalize score calculations of existing classifiers
• provide publicly available dataset based accuracy and speed tests
• more Lucene based classification algorithms

Specific subtasks for each of the above topics should be created to discuss each of them in depth.

Sub-Tasks

• Evaluate Lucene classification on publicly available datasets: Resolved
• Lucene classification score calculation normalize and return lists: Closed
• Separate the classifiers to online and caching where possible: Closed