Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16559

Parquet schema evolution for partitioned tables may break if table and partition serdes differ

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Labels:
      None

      Description

      Parquet schema evolution should make it possible to have partitions/tables
      backed by files with different schemas. Hive should match the table columns with file columns based on the column name if possible.
      However if the serde for a table is missing columns from the serde of a partition Hive fails to match the columns together.
      Steps to reproduce:

      CREATE TABLE myparquettable_parted
      (
        name string,
        favnumber int,
        favcolor string,
        age int,
        favpet string
      )
      PARTITIONED BY (day string)
      STORED AS PARQUET;
      
      INSERT OVERWRITE TABLE myparquettable_parted
      PARTITION(day='2017-04-04')
      SELECT
         'mary' as name,
         5 AS favnumber,
         'blue' AS favcolor,
         35 AS age,
         'dog' AS favpet;
      
      alter table myparquettable_parted
      REPLACE COLUMNS
      (
      favnumber int,
      age int
      );   <!--- No cascade option, so the partition will not be altered. 
      

      SELECT * FROM myparquettable_parted where day='2017-04-04';
      will fail with:
      java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.io.IntWritable

      Hive should either match the columns together or prevent the user from dropping columns from the table.

        Attachments

        1. HIVE-16559.06.patch
          4 kB
          Barna Zsombor Klara
        2. HIVE-16559.05.patch
          5 kB
          Barna Zsombor Klara
        3. HIVE-16559.04.patch
          5 kB
          Barna Zsombor Klara
        4. HIVE-16559.03.patch
          12 kB
          Barna Zsombor Klara
        5. HIVE-16559.02.patch
          5 kB
          Barna Zsombor Klara
        6. HIVE-16559.01.patch
          5 kB
          Barna Zsombor Klara

          Issue Links

            Activity

              People

              • Assignee:
                zsombor.klara Barna Zsombor Klara
                Reporter:
                zsombor.klara Barna Zsombor Klara
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: