[OPENNLP-1353] DictonaryLemmatizer missing charset

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.3
• Fix Version/s: 2.0.0
• Component/s: Lemmatizer
• Environment: Windows 10

Language

Java

• Assignee: Unassigned
• Reporter: Robert
• Created: 17/Jan/22 15:35
• Updated: 07/Jun/22 11:58
• Resolved: 07/Jun/22 11:58

Description

The initialization of the DictonaryLemmatizer is not decoding the inputstream correctly due to missing charset.

My dictionary file for the lemmatizer is utf-8 encoded. At DictonaryLemmatizer initialization the system fallback encoding is used because no charset is specified for the InputStreamReader. In my case windows-1252. This leads to the problem that the correct lemmas of words are not found.

E.g. My lemma.dict file contains following line (utf-8):

mäuse      NN     maus   //German word of mice

And the InputStreamReader decodes it as windows-1252:

mäuse    NN    maus