[OPENNLP-484] Improve features related to abbreviation dictionary in Tokenizer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tools-1.5.3
• Fix Version/s: tools-1.5.3
• Component/s: Tokenizer
• Assignee: William Colen
• Reporter: William Colen
• Created: 27/Mar/12 16:50
• Updated: 08/May/12 19:23
• Resolved: 08/May/12 19:23

Description

I noticed, after doing some benchmark, that we can improve Tokenizer effectiveness by changing a little how features related with the abbreviation dictionary are collected.