[SPARK-27733] Upgrade to Avro 1.10.1

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: Build, SQL
• Assignee: Ismaël Mejía
• Reporter: Ismaël Mejía
• Created: 15/May/19 21:00
• Updated: 12/Dec/22 18:11
• Resolved: 20/Jan/21 23:43

Description

Avro 1.9.2 was released with many nice features including reduced size (1MB less), and removed dependencies, no paranamer, no shaded guava, security updates, so probably a worth upgrade.

Avro 1.10.0 was released and this is still not done.

There is at the moment (2020/08) still a blocker because of Hive related transitive dependencies bringing older versions of Avro, so we could say that this is somehow still blocked until HIVE-21737 is solved.

Issue Links

• blocks: SPARK-26346 Upgrade parquet to 1.11.1
• is related to: SPARK-34557 Exclude Avro's transitive zstd-jni dependency
• is related to: SPARK-34778 Upgrade to Avro 1.10.2
• is related to: HIVE-21737 Upgrade Avro to version 1.10.2
• relates to: AVRO-2400 Avro 1.9.0 can't resolve schemas that can be resolved in 1.8.2
• relates to: AVRO-2817 Avro file generated using avro-1.8.2 is not readable from avro-1.9.2 for certian type of schemas
• relates to: SPARK-20202 Remove references to org.spark-project.hive
• links to: [Github] Pull Request #31232 (iemejia)
• links to: [Github] Pull Request #31866 (iemejia)
• links to: GitHub Pull Request #27609