[SEDONA-150] Drop Spark 2.4 and Scala 2.11 support

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Jia Yu
• Created: 22/Aug/22 03:53
• Updated: 22/Oct/22 21:42
• Resolved: 22/Oct/22 21:42

Description

We will drop Spark 2.4 and Scala 2.11 support from Sedona 1.3.0-incubating. See the discussion on Sedona dev mailing list: https://lists.apache.org/thread/h7qhx8b7ljjkyhb0gy9ydvjyym80fgo1

 

The following change will be made:

1. Remove Spark 2.4 and Scala 2.11 compilation configuration from the main pom.xml
2. Remove Spark 2.4 and Scala 2.11 CI from GitHub Action: Java, Python and R
3. New release will include Scala 2.13 support for Spark
4. Remove Spark 2.4 and Scala 2.11 compilation instruction from Sedona website

 

The following content will be preserved but might be changed without notice

1. The source code compatibility with Spark 2.4 and Scala 2.11
2. The Python script "spark-version-converter.py" to convert Sedona source code from Spark 3.x to Spark 2.4

 

In Sedona 1.3.0-incubating, the following jars will be released to Maven Central

1. seodna-common
2. sedona-core-3.0_2.12
3. sedona-sql-3.0_2.12
4. sedona-viz-3.0_2.12
5. sedona-python-adapter-3.0_2.12
6. sedona-flink_2.12
7. sedona-core-3.0_2.13
8. sedona-sql-3.0_2.13
9. sedona-viz-3.0_2.13
10. sedona-python-adapter-3.0_2.13

Issue Links

• is related to: SEDONA-128 Change Sedona python code to support pyspark 3.3
• links to: GitHub Pull Request #671