[OPENNLP-1405] Enhance JavaDoc in opennlp.tools.tokenize package

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.1
• Component/s: Documentation, Tokenizer
• Assignee: Martin Wiesner
• Reporter: Martin Wiesner
• Created: 04/Dec/22 17:40
• Updated: 10/Dec/22 14:22
• Resolved: 10/Dec/22 14:22

Description

The JavaDoc the opennlp.tools.tokenize package suffers from several inconsistencies and missing descriptions. Moreover, several typos are present that need sanitizing.

It needs enhancements and/or additions to provide more clarity for readers.