[SEDONA-276] Add support for Spark 3.4

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.1
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Martin Andersson
• Created: 17/Apr/23 14:13
• Updated: 04/Jun/23 02:01
• Resolved: 04/Jun/23 02:01

Description

Spark 3.4.0 was announced on April 14. https://lists.apache.org/thread/o1o9n1t036rmfp45q1jrjs0zzzx0j156

Findings from a quick test build using -Dspark.version=3.4.0:

1. Incompatible logging api versions causes test failures.
2. GeoParquet and Geotiff datasources fails to compile due to api changes in Spark.

Setting log4j and slf4j to the same versions as Spark 3.4.0 uses solves 1. I don't know how that will affect other Spark versions.

To solve 2 we might need to use reflection to handle the api changes in Spark.

Issue Links

• incorporates: SEDONA-288 Transition Sedona logs to slf4j
• links to: GitHub Pull Request #825