[SPARK-39775] Regression due to AVRO-2035

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.1, 3.2.3, 3.4.0
• Component/s: Spark Core
• Assignee: Yuming Wang
• Reporter: Nandini
• Created: 14/Jul/22 06:59
• Updated: 05/Aug/22 03:27
• Resolved: 05/Aug/22 03:27

Description

With the upgrade in Avro version to 1.9.0, for schema evolution Avro added https://issues.apache.org/jira/browse/AVRO-2035(enable validation of default values in schemas by default) which is causing regressions when user upgrades their Spark verion.

Repro code:

import org.apache.spark.sql.avro.functions._
val avroTypeStruct = s"""
                        |{
                        |  "type": "record",
                        |  "name": "struct",
                        |  "fields": [
                        |    {"name": "id", "type": "long", "default": null}
                        |  ]
                        |}""".stripMargin

val df = spark.range(10).select(struct('id).as("struct"))
val avroStructDF = df.select(to_avro('struct, avroTypeStruct).as("avro"))
avroStructDF.select(from_avro('avro, avroTypeStruct)).show()

Hive mitigated it by disabling this feature altogether in https://issues.apache.org/jira/browse/HIVE-24797
Spark-Hive integration also imported the above changes in https://issues.apache.org/jira/browse/SPARK-34512

Can we have a fix for all the senarios?

Issue Links

• links to: [Github] Pull Request #37191 (wangyum)