[HIVE-12475] Parquet schema evolution within array<struct<>> doesn't work

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: File Formats
• Assignee: Mohammad Islam
• Reporter: Mohammad Islam
• Created: 20/Nov/15 00:25
• Updated: 16/Feb/16 23:50
• Resolved: 02/Dec/15 15:35

Description

If we create a table with type array<struct<>>, and later added a field in the struct, we got the following exception.

The following SQL statements would recreate the error:

CREATE TABLE pq_test (f1 array<struct<c1:int,c2:int>>) STORED AS PARQUET;
INSERT INTO TABLE pq_test select array(named_struct("c1",1,"c2",2)) FROM tmp LIMIT 2;

SELECT * from pq_test;

ALTER TABLE pq_test REPLACE COLUMNS (f1 array<struct<c1:int,c2:int,cccccc:int>>); //***** cccccc
SELECT * from pq_test;

Exception:

Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.getStructFieldData(ArrayWritableObjectInspector.java:142)
at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:363)
at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:316)
at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:199)
at org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:61)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.doSerialize(LazySimpleSerDe.java:236)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.serialize(AbstractEncodingAwareSerDe.java:55)
at org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:71)
at org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:40)
at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:89)