Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6835

Reading of partitioned Avro data fails if partition schema does not match table schema

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      To reproduce:

      create table testarray (a array<string>);
      
      load data local inpath '/home/ahsu/test/array.txt' into table testarray;
      
      # create partitioned Avro table with one array column
      create table avroarray partitioned by (y string) row format serde 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type": "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"} } ] }')  STORED as INPUTFORMAT  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'  OUTPUTFORMAT  'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
      
      insert into table avroarray partition(y=1) select * from testarray;
      
      # add an int column with a default value of 0
      alter table avroarray set serde 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type": "record", "fields": [ {"name":"intfield","type":"int","default":0},{ "name":"a", "type":{"type":"array","items":"string"} } ] }');
      
      # fails with ClassCastException
      select * from avroarray;
      

      The select * fails with:

      Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
      

        Attachments

        1. HIVE-6835.5.patch
          121 kB
          Anthony Hsu
        2. HIVE-6835.4.patch
          116 kB
          Anthony Hsu
        3. HIVE-6835.3.patch
          47 kB
          Anthony Hsu
        4. HIVE-6835.2.patch
          44 kB
          Anthony Hsu
        5. HIVE-6835.1.patch
          42 kB
          Anthony Hsu

          Issue Links

            Activity

              People

              • Assignee:
                erwaman Anthony Hsu
                Reporter:
                erwaman Anthony Hsu
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: