[PARQUET-1655] [C++] Decimal comparisons used for min/max statistics are not correct

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: cpp-4.0.0
• Component/s: parquet-cpp
• Labels: pull-request-available
• Assignee: Micah Kornfield
• Reporter: Philip Felton
• Created: 06/Aug/19 15:39
• Updated: 23/Mar/21 20:16
• Resolved: 04/Mar/21 16:18

Description

The Parquet Format specifications says

If the column uses int32 or int64 physical types, then signed comparison of the integer values produces the correct ordering. If the physical type is fixed, then the correct ordering can be produced by flipping the most-significant bit in the first byte and then using unsigned byte-wise comparison.

However this isn't followed in the C++ Parquet code. 16-byte decimal comparison is implemented using a lexicographical comparison of signed chars.

This appears to be because the function https://github.com/apache/arrow/blob/master/cpp/src/parquet/statistics.cc#L183 just goes off the sort_order (signed) and physical_type (FIXED_LENGTH_BYTE_ARRAY), there is no override for decimal.

Issue Links

• is duplicated by: ARROW-12054 [C++] Parquet statistics incorrect for decimal128
• links to: GitHub Pull Request #9582