[TIKA-3147] Strip punctuation in lang id component within tika-eval

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.25
• Assignee: Unassigned
• Reporter: Tim Allison
• Created: 24/Jul/20 21:17
• Updated: 27/Jul/20 20:58
• Resolved: 27/Jul/20 17:26

Description

I noticed that "the quick brown fox jumped over the lazy dog" was identified as English in tika-eval. However, if I added semi-colons, it was identified as Chinese.

This is in alignment with what I've recently seen on a new batch of technical documents that are mostly numbers and abbreviations...these are being identified as Chinese.

Ideally, we'd strip out punctuation while building the models, and while running language id.

As a short term fix, we should at least strip out punctuation during detection.