[SEDONA-67] Support Spark 3.2

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.1
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Adam Binford
• Created: 18/Oct/21 14:41
• Updated: 12/Dec/21 20:22
• Resolved: 12/Dec/21 20:22

Description

An initial attempt to build master with Spark 3.2 revealed at least two things that need to be addressed:

• The API for registering SQL functions changed (https://github.com/apache/spark/pull/32587)
• A change in TreeNode added a new required function `withNewChildrenInternal` that all the expressions are now missing (https://github.com/apache/spark/pull/32030)

Need to figure out if it's possible to fix these in a backward compatible way so it still supports all 3.0+ versions, try to create a separate release for 3.2+, or only support the latest Spark version.

Issue Links

• links to: GitHub Pull Request #557
• links to: GitHub Pull Request #561