[HIVE-16559] Parquet schema evolution for partitioned tables may break if table and partition serdes differ

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Serializers/Deserializers
• Assignee: Barna Zsombor Klara
• Reporter: Barna Zsombor Klara
• Created: 28/Apr/17 10:06
• Updated: 23/May/18 00:00
• Resolved: 27/Jun/17 15:22

Description

Parquet schema evolution should make it possible to have partitions/tables
backed by files with different schemas. Hive should match the table columns with file columns based on the column name if possible.
However if the serde for a table is missing columns from the serde of a partition Hive fails to match the columns together.
Steps to reproduce:

CREATE TABLE myparquettable_parted
(
  name string,
  favnumber int,
  favcolor string,
  age int,
  favpet string
)
PARTITIONED BY (day string)
STORED AS PARQUET;

INSERT OVERWRITE TABLE myparquettable_parted
PARTITION(day='2017-04-04')
SELECT
   'mary' as name,
   5 AS favnumber,
   'blue' AS favcolor,
   35 AS age,
   'dog' AS favpet;

alter table myparquettable_parted
REPLACE COLUMNS
(
favnumber int,
age int
);   <!--- No cascade option, so the partition will not be altered. 

SELECT * FROM myparquettable_parted where day='2017-04-04';
will fail with:
java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.io.IntWritable

Hive should either match the columns together or prevent the user from dropping columns from the table.

Issue Links

• links to: Review Board