[TIKA-855] Language Detection not working for Japanese and Chinese.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.0
• Component/s: languageidentifier
• Labels: Chinese Japanese

Environment

Windows XP, Vista and Linux Ubuntu 11.10 using Sun Java 6 and Oracle Java 7

• Assignee: Kenneth William Krugler
• Reporter: James Sullivan
• Created: 02/Feb/12 00:26
• Updated: 20/Feb/12 10:18
• Resolved: 02/Feb/12 00:58

Description

I have tried Tika 1.0 language detection (java -jar tika.jar -l .\Japanese.txt) on several Japanese files (both PDF and text files) and it consistently returns lt (Lithuanian???) instead of ja. I also tried on a Chinese file which similarly incorrectly returned lt. Both English language and French language detection worked correctly.

Issue Links

• depends upon: TIKA-856 Support CJK (Chinese, Japanese and Korean) language detection