[TIKA-2933] Revisit "replacement" encoding mappings in StandardHtmlEncodingDetector.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Tim Allison
• Created: 30/Aug/19 12:08
• Updated: 09/Sep/19 11:04

Description

Revisit "replacement" encoding mappings in StandardHtmlEncodingDetector.

I'm finally getting around to running the comparisons between our legacy HTMLEncodingDetector and the newer StandardHTMLEncodingDetector. More analysis is required, but the newer one is, generally better*. One area for improvement/explanation, though is in the "replacement" encoding.

• There are 1 million more "common words" in text extracted from files with the StandardHtmlEncodingDetector than with only our legacy. There are 133M common words in our legacy extracts so that's less than 1% improvement.

Issue Links

• is related to: TIKA-2673 HtmlEncodingDetector doesn't follow the specification
• relates to: TIKA-2937 Improve legacy HTML charset detector by replicating Standard's behavior for UTF-16
• relates to: TIKA-2940 Consider an ensemble charset detection method