[OPENNLP-215] Add Tokenizer Training API section

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: tools-1.5.3
• Component/s: Documentation
• Assignee: Aliaksandr Autayeu
• Reporter: Jörn Kottmann
• Created: 11/Jul/11 12:28
• Updated: 01/Dec/11 20:38
• Resolved: 01/Dec/11 20:38

Description

The documentation is lacking a section for the tokenizer which explains how the training api can be used. This section should be written in a similar fashion as in other components, e.g. sentence detector or name finder.