[HIVE-6835] Reading of partitioned Avro data fails if partition schema does not match table schema

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.14.0
• Assignee: Anthony Hsu
• Reporter: Anthony Hsu
• Created: 03/Apr/14 17:09
• Updated: 13/Nov/14 19:43
• Resolved: 25/Apr/14 20:57

Description

To reproduce:

create table testarray (a array<string>);

load data local inpath '/home/ahsu/test/array.txt' into table testarray;

# create partitioned Avro table with one array column
create table avroarray partitioned by (y string) row format serde 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type": "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"} } ] }')  STORED as INPUTFORMAT  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'  OUTPUTFORMAT  'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';

insert into table avroarray partition(y=1) select * from testarray;

# add an int column with a default value of 0
alter table avroarray set serde 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type": "record", "fields": [ {"name":"intfield","type":"int","default":0},{ "name":"a", "type":{"type":"array","items":"string"} } ] }');

# fails with ClassCastException
select * from avroarray;

The select * fails with:

Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector

Issue Links

• is related to: HIVE-3953 Reading of partitioned Avro data fails because of missing properties
• is related to: HIVE-6131 New columns after table alter result in null values despite data
• relates to: HIVE-8560 SerDes that do not inherit AbstractSerDe do not get table properties during initialize()