[OPENNLP-485] Improve the AD NameSample formater

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tools-1.5.3
• Fix Version/s: tools-1.5.3
• Component/s: Formats
• Assignee: William Colen
• Reporter: William Colen
• Created: 27/Mar/12 16:59
• Updated: 08/May/12 19:26
• Resolved: 08/May/12 19:26

Description

We need to improve how contractions are handled: some are expanded to more than 2 tokens. Also should force tokenization of named entities that has punctuations.