[NUTCH-1733] parse-html to support HTML5 charset definitions

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8, 2.2.1
• Fix Version/s: 2.3, 1.9
• Component/s: parser
• Assignee: Unassigned
• Reporter: Sebastian Nagel
• Created: 11/Mar/14 10:55
• Updated: 01/May/14 06:22
• Resolved: 09/Apr/14 22:08

Description

HTML 5 allows to specify the character encoding of a page per

• <meta charset="...">
• Unicode Byte Order Mark (BOM)

These are allowed in addition to previous HTTP/http-equiv Content-Type, see [1.

Parse-html ignores both meta charset and BOM, falls back to the default encoding (cp1252). Parse-tika sets the encoding appropriately.