[ORC-222] StringStatisticsImpl munges min/max during write

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Component/s: encoding
• Assignee: Unassigned
• Reporter: Dain Sundstrom
• Created: 08/Aug/17 00:21
• Updated: 18/Jan/18 16:48
• Resolved: 18/Jan/18 16:48

Description

String statistics are collected using Text which compares using raw bytes which are assumed to be UTF-8. When the input contains invalid UTF-8 sequences and these are the min or max value, the writer converts these invalid sequences to a java.lang.String, which replaces invalid UTF-8 sequences with the replacement character (0xFFFD). This conversion happens here:

https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java#L611

To work around this issue, the writer should use the `setMinimumBytes` Protocol Buffers API instead.

The same issue exists during read, where the bytes are round tripped through java.lang.String. The read code is here:

https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java#L528