[HUDI-2811] Support Spark 3.2 and Parquet 1.12.x

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Done
• Fix Version/s: 0.11.0
• Component/s: spark
• Labels: pull-request-available sev:critical
• Assignee: Yann Byron
• Reporter: Raymond Xu

Reviewers

Raymond Xu

• Created: 20/Nov/21 20:00
• Updated: 24/Jan/22 07:29
• Resolved: 11/Jan/22 22:05

Description

Reported issues

• https://github.com/apache/hudi/issues/4001
• https://github.com/apache/hudi/issues/3841
• https://github.com/apache/hudi/issues/4202
• https://github.com/apache/hudi/issues/3834
•  

Issue Links

• blocks: HUDI-864 parquet schema conflict: optional binary <some-field> (UTF8) is not a group
• blocks: HUDI-1079 Cannot upsert on schema with Array of Record with single field
• blocks: HUDI-2323 Upsert of Case Class with single field causes SchemaParseException
• blocks: HUDI-2899 Fix DataFormatter usages removed in Spark 3.2
• blocks: HUDI-2928 Evaluate rebasing Hudi's default compression from Gzip to Zstd
• blocks: HUDI-3088 Make Spark 3 the default profile for build and test
• depends upon: HUDI-2781 Test 0.10 RC for Spark 3.x
• is blocked by: HUDI-2896 Exception is thrown from HoodieAvroUtils.getNestedFieldVal() with returnNullIfNotFound as true
• is duplicated by: HUDI-2812 Upgrade Parquet to 1.12
• is duplicated by: HUDI-2594 Upgrade parquet version 1.12.1
• relates to: HUDI-2929 Spark Datasource read issue in Spark 3.2.0
• relates to: HUDI-3131 Spark3.1.1 CTAS error
• links to: GitHub Pull Request #4270