[SPARK-39775] Regression due to AVRO-2035 - ASF JIRA

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.2.1
Fix Version/s: 3.3.1, 3.2.3, 3.4.0
Component/s: Spark Core
Labels:
None

Description

With the upgrade in Avro version to 1.9.0, for schema evolution Avro added https://issues.apache.org/jira/browse/AVRO-2035(enable validation of default values in schemas by default) which is causing regressions when user upgrades their Spark verion.

Repro code:

import org.apache.spark.sql.avro.functions._
val avroTypeStruct = s"""
                        |{
                        |  "type": "record",
                        |  "name": "struct",
                        |  "fields": [
                        |    {"name": "id", "type": "long", "default": null}
                        |  ]
                        |}""".stripMargin

val df = spark.range(10).select(struct('id).as("struct"))
val avroStructDF = df.select(to_avro('struct, avroTypeStruct).as("avro"))
avroStructDF.select(from_avro('avro, avroTypeStruct)).show()