[IMPALA-7087] Impala is unable to read Parquet decimal columns with lower precision/scale than table metadata

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Component/s: Backend
• Labels: decimal parquet ramp-up
• Target Version: Product Backlog
• Epic Color: ghx-label-6
• Assignee: Zoltán Borók-Nagy
• Reporter: Tim Armstrong
• Created: 29/May/18 16:11
• Updated: 02/Nov/21 12:51
• Resolved: 14/Jul/21 13:14

Description

This is similar to IMPALA-2515, except relates to a different precision/scale in the file metadata rather than just a mismatch in the bytes used to store the data. In a lot of cases we should be able to convert the decimal type on the fly to the higher-precision type.

ERROR: File '/hdfs/path/000000_0_x_2' column 'alterd_decimal' has an invalid type length. Expecting: 11 len in file: 8

It would be convenient to allow reading parquet files where the precision/scale in the file can be converted to the precision/scale in the table metadata without loss of precision.

Issue Links

• breaks: IMPALA-10808 Crash of illegal decimal schema in test_fuzz_decimal_tbl
• is related to: IMPALA-6373 Allow type widening primitive conversion on parquet tables