[HUDI-3679] Spark Compatibility Model

• Type: Epic
• Status: Closed
• Priority: Blocker
• Resolution: Done
• Fix Version/s: 0.11.0
• Component/s: dependencies
• Epic Name: Spark Compatibility
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 22/Mar/22 15:51
• Updated: 09/May/22 09:08
• Resolved: 09/May/22 09:08

Description

Currently we have inconsistent models of aliasing Spark version in the bundle name ("spark3", "spark3.1.x", "spark3.2.1", etc).

To avoid confusion in the community, and make things consistent following compatibility pledge is proposed: 

Hudi bundles will stay compatible with all patch versions of Spark w/in single minor version branch (ie Spark 3.1.x, 3.2.x, etc)

 

Such compatibility model will be correspondingly reflected in the bundle names: corresponding spark minor version branch will be aliased in the bundle name to signal its compatibility with Spark versions w/in such branch ("spark3.1" for Spark 3.1.x, etc)

To transition to the new model following steps are proposed:

1. Add new artifacts carrying Spark minor versions in the bundle names (for ex, "spark3.2", "spark3.1", "spark2.4") in 0.11
2. Deprecate single and triple digit bundles ("spark3", "spark2", "spark3.2.1") in favor of double-digit in 0.11
3. Discontinue deprecated bundles (spark3, spark2, etc) from 0.12

 

As such following bundles will be released in 0.11

• "spark2.4" (Scala 2.11/2.12)
• "spark3.1"
• "spark3.2"
• "spark3" (this is the same as "spark3.2")
• "spark2" (Scala 2.11/2.12, this is the same as "spark2.4")