[NUTCH-2254] Charset issues when using -addBinaryContent and -base64 options

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.11
• Component/s: indexer
• Assignee: Sebastian Nagel
• Reporter: Federico Bonelli
• Created: 21/Apr/16 12:36
• Updated: 28/Jan/21 14:04
• Resolved: 27/Apr/16 20:59

Description

The bug is reproducible with these steps:

1. find a site with cp1252 encoded pages like "http://www.ilsole24ore.com/" and characters with accents (byte representation >127, like [àèéìòù])
2. start a crawl on that site indexing on Solr with options -addBinaryContent -base64
3. find a document inside the newly indexed Solr collection with those accented characters
4. get the base64 binary representation for said html page and decode it back to raw binary, save it

The file obtained will have invalid characters, which are neither UTF-8 nor cp1252.

Issue Links

• is related to: NUTCH-1807 avoid methods relying on system-specific default locale / charset
• is related to: NUTCH-1785 Ability to index raw content