[OPENNLP-237] Add abbreviation dictionary support to Tokenizer

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: tools-1.5.2-incubating
• Fix Version/s: tools-1.5.2-incubating
• Component/s: Tokenizer
• Assignee: William Colen
• Reporter: William Colen
• Created: 22/Jul/11 16:36
• Updated: 27/Jul/11 14:04
• Resolved: 27/Jul/11 14:04

Description

The Tokenizer component can take advantage of using an abbreviation dictionary in context generator.
Although it modifies the default tokenizer context generator it won't break compatibility with old models because the features would be applied only if the dictionary is present.