Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12475

Parquet schema evolution within array<struct<>> doesn't work

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: File Formats
    • Labels:
      None

      Description

      If we create a table with type array<struct<>>, and later added a field in the struct, we got the following exception.

      The following SQL statements would recreate the error:

      CREATE TABLE pq_test (f1 array<struct<c1:int,c2:int>>) STORED AS PARQUET;
      INSERT INTO TABLE pq_test select array(named_struct("c1",1,"c2",2)) FROM tmp LIMIT 2;

      SELECT * from pq_test;

      ALTER TABLE pq_test REPLACE COLUMNS (f1 array<struct<c1:int,c2:int,cccccc:int>>); //***** cccccc
      SELECT * from pq_test;

      Exception:

      Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
      at org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.getStructFieldData(ArrayWritableObjectInspector.java:142)
      at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:363)
      at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:316)
      at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:199)
      at org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:61)
      at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.doSerialize(LazySimpleSerDe.java:236)
      at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.serialize(AbstractEncodingAwareSerDe.java:55)
      at org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:71)
      at org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:40)
      at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:89)

        Attachments

        1. HIVE-12475.1.patch
          4 kB
          Sergio Peña

          Activity

            People

            • Assignee:
              kamrul Mohammad Kamrul Islam
              Reporter:
              kamrul Mohammad Kamrul Islam
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: