[TIKA-369] Improve accuracy of language detection

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6
• Component/s: languageidentifier
• Assignee: Kenneth William Krugler
• Reporter: Kenneth William Krugler
• Created: 24/Jan/10 18:52
• Updated: 15/Dec/21 16:02
• Resolved: 15/Dec/21 16:02

Description

Currently the LanguageProfile code uses 3-grams to find the best language profile using Pearson's chi-square test. This has three issues:

1. The results aren't very good for short runs of text. Ted Dunning's paper (attached) indicates that a log-likelihood ratio (LLR) test works much better, which would then make language detection faster due to less text needing to be processed.
2. The current LanguageIdentifier.isReasonablyCertain() method uses an exact value as a threshold for certainty. This is very sensitive to the amount of text being processed, and thus gives false negative results for short runs of text.
3. Certainty should also be based on how much better the result is for language X, compared to the next best language. If two languages both had identical sum-of-squares values, and this value was below the threshold, then the result is still not very certain.

Issue Links

• is duplicated by: TIKA-1091 Class LanguageIdentifier wrongly detecting the english language sentance
• is related to: NUTCH-666 Analysis plugins for multiple language and new Language Identifier Tool
• is superceded by: TIKA-1723 Integrate language-detector into Tika
• relates to: TIKA-209 Language detection is weak.
• relates to: TIKA-496 Language identifier profile comparison favors large profiles
• relates to: TIKA-322 Improve encoding detection speed and accuracy
• relates to: TIKA-1723 Integrate language-detector into Tika
• relates to: TIKA-465 LanguageIdentifier API enhancements