[HUDI-3662] Make sure CI tests against all supported Spark versions

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Duplicate
• Component/s: tests-ci
• Story Points: 4
• Epic Link: CI tests improvements
• Assignee: Shiyan Xu
• Reporter: Alexey Kudinkin
• Created: 18/Mar/22 22:56
• Updated: 28/Apr/22 14:19
• Resolved: 28/Apr/22 14:19

Description

Currently we only run CI against a single supported version (which is 2.4.4) this is not helpful in giving us confidence that we stay compatible across all of the Spark versions that we've committed to (2.4.4, 3.1.x, 3.2.x).

 

To address that we should make sure that we run at least subset of tests against all versions of Spark that we pledged to support.

Issue Links

• duplicates: HUDI-3687 Make sure CI run tests against all Spark versions
• relates to: HUDI-3549 Investigate spark3 read issues w/ hudi spark bundle 3.2 with S3 dataset