[HIVE-24797] Disable validate default values when parsing Avro schemas

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.9, 4.0.0-alpha-1
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 19/Feb/21 10:06
• Updated: 17/Nov/22 08:48
• Resolved: 24/Feb/21 06:53

Description

It will throw exceptions when upgrading Avro to 1.10.1 for this schema:

{
    "type": "record",
    "name": "EventData",
    "doc": "event data",
    "fields": [
        {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items": "string"}, "default": null }
    ]
}

org.apache.avro.AvroTypeException: Invalid default for field ARRAY_WITH_DEFAULT: null not a {"type":"array","items":"string"}
	at org.apache.avro.Schema.validateDefault(Schema.java:1571)
	at org.apache.avro.Schema.access$500(Schema.java:87)
	at org.apache.avro.Schema$Field.<init>(Schema.java:544)
	at org.apache.avro.Schema.parse(Schema.java:1678)
	at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
	at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
	at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
	at org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)

Issue Links

• is related to: AVRO-2817 Avro file generated using avro-1.8.2 is not readable from avro-1.9.2 for certian type of schemas
• is related to: AVRO-2035 enable validation of default values in schemas by default
• relates to: SPARK-34512 Disable validate default values when parsing Avro schemas
• links to: GitHub Pull Request #1994
• links to: GitHub Pull Request #2670
• links to: GitHub Pull Request #2699