[IMPALA-6373] Allow type widening primitive conversion on parquet tables

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0
• Fix Version/s: Impala 3.1.0
• Component/s: Backend
• Labels: parquet
• Epic Color: ghx-label-7
• Assignee: Fredy Wijaya
• Reporter: Patrick Bayne
• Created: 06/Jan/18 23:24
• Updated: 23/Aug/18 21:22
• Resolved: 23/Aug/18 21:22

Description

This improvement is very useful for users who have schema evolution on their tables, or users that use Impala and Hive for querying data and are allowing for this. Beings this improvement is in Hive, a Hive table with a "bigint" can read parquet files with "int32" and "int64" types. This same query will fail in Impala due to the unexpected data type, with an error similar to: "Column type: BIGINT, Parquet schema: optional int32"

Hive 2.x's addition of type widening is found in the following JIRAs:
https://issues.apache.org/jira/browse/HIVE-12080
https://issues.apache.org/jira/browse/HIVE-14085

Here's a list of widening conversions the upstream Hive improvement supports:

tinyint -> smallint,int,bigint,float,double
smallint -> int,bigint,float,double
int -> bigint,float,double
bigint -> float,double
float -> double
double -> –

Issue Links

• relates to: IMPALA-7087 Impala is unable to read Parquet decimal columns with lower precision/scale than table metadata