Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6373

Allow type widening primitive conversion on parquet tables

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 2.10.0
    • Impala 3.1.0
    • Backend
    • ghx-label-7

    Description

      This improvement is very useful for users who have schema evolution on their tables, or users that use Impala and Hive for querying data and are allowing for this. Beings this improvement is in Hive, a Hive table with a "bigint" can read parquet files with "int32" and "int64" types. This same query will fail in Impala due to the unexpected data type, with an error similar to: "Column type: BIGINT, Parquet schema: optional int32"

      Hive 2.x's addition of type widening is found in the following JIRAs:
      https://issues.apache.org/jira/browse/HIVE-12080
      https://issues.apache.org/jira/browse/HIVE-14085

      Here's a list of widening conversions the upstream Hive improvement supports:

      tinyint -> smallint,int,bigint,float,double
      smallint -> int,bigint,float,double
      int -> bigint,float,double
      bigint -> float,double
      float -> double
      double -> –

      Attachments

        Issue Links

          Activity

            People

              fredyw Fredy Wijaya
              SombreroPatrick Patrick Bayne
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: