[OPENNLP-481] ADTokenSampleStream is not handling hyphenated tokens correctly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tools-1.5.3
• Fix Version/s: tools-1.5.3
• Component/s: Formats
• Assignee: William Colen
• Reporter: William Colen
• Created: 23/Mar/12 00:34
• Updated: 13/Jul/12 03:57
• Resolved: 13/Jul/12 03:57

Description

Hyphenated tokens are not handled, for example "aprova-se" should be separated in two tokens. ADTokenSampleStream relies on a DictionaryDetokenizer, that don't know that "aprova-" "se" should be merged.