[HUDI-2781] Test 0.10 RC for Spark 3.x

• Type: Test
• Status: Closed
• Priority: Blocker
• Resolution: Done
• Fix Version/s: 0.10.0
• Component/s: spark
• Labels: pull-request-available sev:high
• Assignee: Yann Byron
• Reporter: Shiyan Xu

Reviewers

Shiyan Xu

• Created: 17/Nov/21 07:55
• Updated: 21/Dec/21 04:44
• Resolved: 14/Dec/21 02:23

Description

Combinations

1. Spark 3.0 & 3.1.x against Hive 2
2. Spark 3.0 & 3.1.X against Hive 3
3. Spark 3.2 against Hive 2

 
Let's test a COW and MOR long running DAG across these environments and get a report with bugs/issues
 
We have YAMLs here, that can be run across all different environments listed here.
https://github.com/apache/hudi/tree/master/docker/demo/config/test-suite

Issue Links

• Discovered while testing: HUDI-1711 Avro Schema Exception with Spark 3.0 in 0.7
• is depended upon by: HUDI-2811 Support Spark 3.2 and Parquet 1.12.x
• links to: GitHub Pull Request #4085
• links to: GitHub Pull Request #4138