[SEDONA-212] Move shading to separate maven modules

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Martin Andersson
• Created: 09/Dec/22 09:28
• Updated: 04/Jan/23 22:40
• Resolved: 04/Jan/23 22:40

Description

Shading makes it impossible for mavens dependency resolution mechanism to work since dependencies are stripped from the pom.xml and bundled with the jar file. Shading can lead to unpredictable behavior with multiple versions or wrong versions of dependencies on the classpath. Those errors are usually hard to track down and fix for end users.

A few examples: 

• In SEDONA-210 if shading wasn't involved scala-collection-compat could be excluded by the user in sbt.
• We see the same problem with org.datasyslab.sernetcdf, although not caused by Sedona. In it's shaded jar it pulls in an old version of hadoop. Sedona also pulls in hadoop. With two version on the classpath we see unpredictable behavior.
• Adding geotools to a Sedona project is hard without the, out of tree, geotoolswrapper. Geotools pulls in JTS. If the JTS version shaded into Sedona jars differs from the version pulled by geotools it's a coin toss which version is used.

 

I propose we move shading from the parent pom.xml to two separate modules: sedona-spark-shaded and sedona-flink-shaded. In Python and R environments that don't have access to a full jvm build tool (like mvn, gradle and sbt) the shaded jars can be used as a convenience. In jvm projects it probably makes more sense to use the non shaded dependencies lika sedona-sql and sedona-flink.

 

Issue Links

• links to: GitHub Pull Request #735