[SPARK-38077] Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.2.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Nadav Samet
• Created: 31/Jan/22 13:40
• Updated: 16/Apr/22 21:09
• Resolved: 16/Apr/22 21:09

Description

PR 35243 introduced a new parameter to class `Invoke` with a default value (`isDeterministic: Boolean = true`). Existing Spark libraries (such as frameless) that invoke Invoke directly expect a method with 7 parameters, and the new version expects 8. If Frameless would recompile with Spark 3.2.1, the updated library will NOT be binary compatible with Spark 3.2.0. Adding default parameters to existing methods should be avoided.

One way forward would be to revert the change in the constructor and introduce a second constructor or a companion method that takes all the 8 parameters.

Issue Links

• is broken by: SPARK-37957 Deterministic flag is not handled for V2 functions
• links to: [Github] Pull Request #35378 (sunchao)