[HUDI-7915] Spark 4 support

• Type: New Feature
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Labels: pull-request-available
• Story Points: 30
• Epic Link: Enhancing Spark integration in Hudi 1.0
• Assignee: Shawn Chang
• Reporter: Ethan Guo (this is the old account; please use "yihua")
• Created: 21/Jun/24 23:32
• Updated: 13/Oct/24 22:53

Description

Spark 4.0.0-preview1 is out.  We should start integrating Hudi with Spark 4 and surface any issues early on.

https://spark.apache.org/news/spark-4.0.0-preview1.html

Issue Links

• relates to: HUDI-8086 Get rid of logic not needed by spark3.3+
• links to: GitHub Pull Request #11517
• links to: GitHub Pull Request #11539