[OPENNLP-941] Add eval support to detokenizer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.9.0
• Component/s: Tokenizer
• Labels: help-wanted
• Assignee: Unassigned
• Reporter: Jörn Kottmann
• Created: 15/Jan/17 17:01
• Updated: 16/Dec/22 12:40
• Resolved: 16/Dec/22 12:40

Description

It would be nice to be able to evaluate a detokenizer on tokenizer training data.

Issue Links

• links to: GitHub Pull Request #308