Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6373

Allow type widening primitive conversion on parquet tables

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: Impala 3.1.0
    • Component/s: Backend
    • Labels:
    • Epic Color:
      ghx-label-7

      Description

      This improvement is very useful for users who have schema evolution on their tables, or users that use Impala and Hive for querying data and are allowing for this. Beings this improvement is in Hive, a Hive table with a "bigint" can read parquet files with "int32" and "int64" types. This same query will fail in Impala due to the unexpected data type, with an error similar to: "Column type: BIGINT, Parquet schema: optional int32"

      Hive 2.x's addition of type widening is found in the following JIRAs:
      https://issues.apache.org/jira/browse/HIVE-12080
      https://issues.apache.org/jira/browse/HIVE-14085

      Here's a list of widening conversions the upstream Hive improvement supports:

      tinyint -> smallint,int,bigint,float,double
      smallint -> int,bigint,float,double
      int -> bigint,float,double
      bigint -> float,double
      float -> double
      double -> –

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                fredyw Fredy Wijaya
                Reporter:
                SombreroPatrick Patrick Bayne
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: