[HIVE-7049] Unable to deserialize AVRO data when file schema and record schema are different and nullable

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Assignee: Daniel Dai
• Reporter: Mohammad Islam
• Created: 12/May/14 21:46
• Updated: 23/Sep/15 17:00
• Resolved: 06/Apr/15 21:06

Description

It mainly happens when
1 )file schema and record schema are not same
2 ) Record schema is nullable but file schema is not.

The potential code location is at class AvroDeserialize

 if(AvroSerdeUtils.isNullableType(recordSchema)) {
      return deserializeNullableUnion(datum, fileSchema, recordSchema, columnType);
    }

In the above code snippet, recordSchema is verified if it is nullable. But the file schema is not checked.

I tested with these values:

recordSchema= ["null","string"]
fielSchema= "string"

And i got the following exception <line numbers might not be the same due to mu debugged code version>.

org.apache.avro.AvroRuntimeException: Not a union: "string" 
        at org.apache.avro.Schema.getTypes(Schema.java:272)
        at org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserializeNullableUnion(AvroDeserializer.java:275)
        at org.apache.hadoop.hive.serde2.avro.AvroDeserializer.worker(AvroDeserializer.java:205)
        at org.apache.hadoop.hive.serde2.avro.AvroDeserializer.workerBase(AvroDeserializer.java:188)
        at org.apache.hadoop.hive.serde2.avro.AvroDeserializer.deserialize(AvroDeserializer.java:174)
        at org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.verifyNullableType(TestAvroDeserializer.java:487)
        at org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.canDeserializeNullableTypes(TestAvroDeserializer.java:407)

Issue Links

• is depended upon by: HIVE-3159 Update AvroSerde to determine schema of new tables