[OPENNLP-697] Tokenizer class is hardcoded in the DocumentSampleStream class.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.6.0
• Fix Version/s: 1.7.1
• Component/s: Doccat, Tokenizer
• Assignee: Unassigned
• Reporter: Praveena B
• Created: 16/May/14 06:27
• Updated: 20/Jan/17 14:25
• Resolved: 20/Jan/17 14:25

Description

While training the DocumentCategorizerME it is possible to set the type of Tokenizer that the categorizer should use.
i,e doccatFactory.setTokenizer(SemicolonTokenizer.INSTANCE);

But the Tokenizer class is hardcoded to WhitespaceTokenizer in the DocumentSampleStream class.
So it is not possible to modify the default tokenizing behaviour even after setting it in the doccatFactory.