[OPENNLP-1202] Word tokenization

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Feedback Received
• Component/s: language model
• Labels: Annotations
• Environment: Windows Server 2016, R version 3.3.3
• Flags: Important
• Assignee: Unassigned
• Reporter: Dippy Aggarwal
• Created: 12/Jun/18 00:30
• Updated: 16/Dec/22 12:30
• Resolved: 16/Dec/22 12:30

Description

Came across an issue for identifying words in a sentence. For words such as can't, the tokenization using openNLP yields two words: "ca" and "n't"

As an example (captured in the screenshot), see the tokenization for the string

When heard the Xenogears soundtrack, so can't really describe.

Note the words marked by ID's 9 and 10 in the openNLP-output.png file. 

Not sure if I am missing any parameters that would produce the correct result? 

Would appreciate any ideas/community's attention to this issue. Thanks.