[TIKA-344] Charset hint in metadata

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.6
• Component/s: parser
• Assignee: Unassigned
• Reporter: Piotr Bartosiewicz
• Created: 07/Dec/09 12:45
• Updated: 13/Dec/09 01:19
• Resolved: 13/Dec/09 01:19

Description

It would be nice if TextParser and HtmlParser support Metadata.CONTENT_ENCODING hint.

In my application I always prefer that hint (if it is present) over the charset detector result, because charset detector is often wrong on short inputs (even if match.confidence is 100) and I know that hint if present is right in 99%.

To be more general, user might be able to change default behaviour by override a function F(hint, detectorResults) -> charset.
Other solution is to create some standard strategies and let user to choose one of them:
a) hint is most important
b) charset detector result is most important
c) create some heuristic using detectorResult.confidence, hint and maybe input length
Maybe the last heuristic method would be good enough for most cases.