[TIKA-334] HtmlParser should use CharsetDetector whenever no charset is specified via meta http-equiv tag

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5
• Fix Version/s: 0.6
• Assignee: Jukka Zitting
• Reporter: Kenneth William Krugler
• Created: 25/Nov/09 18:42
• Updated: 30/Nov/09 01:42
• Resolved: 30/Nov/09 01:42

Description

Currently the HtmlParser will just call TagSoup to parse, without specifying a charset, if no charset is passed in via metadata.

TagSoup uses the platform encoding in this case, which is often going to be wrong.

The right thing to do is to first check for a charset specified by a meta tag. If that doesn't exist, then create a CharsetDetector. If there's a charset in the incoming meta-data, use that to call setDeclaredEncoding().