[TIKA-1723] Integrate language-detector into Tika

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.11
• Fix Version/s: 1.13
• Component/s: languageidentifier
• Flags: Patch
• Assignee: Kenneth William Krugler
• Reporter: Kenneth William Krugler
• Created: 27/Aug/15 23:42
• Updated: 22/Apr/16 22:24
• Resolved: 22/Apr/16 22:24

Description

The language-detector project at https://github.com/optimaize/language-detector is faster, has more languages (70 vs 13) and better accuracy than the built-in language detector.

This is a stab at integrating it, with some initial findings. There are a number of issues this raises, especially if chrismattmann moves forward with turning language detection into a pluggable extension point.

I'll add comments with results below.

Issue Links

• is related to: TIKA-856 Support CJK (Chinese, Japanese and Korean) language detection
• is related to: TIKA-493 Support for macro languages
• is related to: TIKA-491 Add language identification support for Norwegian Bokmål and Norwegian Nynorsk
• is related to: TIKA-492 Add language identification support for North Sami, Lule Sami and South Sami
• is related to: TIKA-568 Language Detection isReasonablyCertain() hides valuable information
• is related to: TIKA-369 Improve accuracy of language detection
• is required by: TIKA-1872 Backport tika-langdetect from 2.x branch to 1.13 branch
• relates to: TIKA-1696 Language Identification with Text Processing Toolkit from MITLL
• relates to: NUTCH-1397 language-identifier incorrectly handles double-barreled language properties
• supercedes: TIKA-496 Language identifier profile comparison favors large profiles
• supercedes: TIKA-856 Support CJK (Chinese, Japanese and Korean) language detection
• supercedes: TIKA-568 Language Detection isReasonablyCertain() hides valuable information
• supercedes: TIKA-369 Improve accuracy of language detection