[TIKA-2256] Japanese character substituted when reading PDF

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.14
• Component/s: parser
• Assignee: Unassigned
• Reporter: Christopher Creutzig
• Created: 30/Jan/17 14:14
• Updated: 22/Jun/17 19:36
• Resolved: 22/Jun/17 19:36

Description

The attached file contains “日本語” in its first line. It was created on Mac OS X 10.11.6 by selecting “Save As PDF” in the system print dialog started from Microsoft Word.

Reading the text from the PDF, the first character is not read as U+65E5, but as U+2F47. Copy & paste from Preview.App results in the correct U+65E5 being copied. (The characters look the same in some fonts, but are different.)

The MATLAB code used for reading looks as follows:

handler = org.apache.tika.sax.ToXMLContentHandler;
parser = org.apache.tika.parser.AutoDetectParser;
metadata = org.apache.tika.metadata.Metadata;
fh = java.io.FileInputStream(fullname);
parser.parse(fh, handler, metadata);
s = handler.toString;