[SPARK-11081] Make spark-core pull in Jersey and javax.ws.rs dependencies separately for easier overriding

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: Build, Spark Core
• Assignee: Unassigned
• Reporter: Mingyu Kim
• Created: 13/Oct/15 07:47
• Updated: 15/Feb/16 20:46
• Resolved: 05/Dec/15 01:18

Description

As seen from this thread (https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCALte62yD8H3=2KVMiFs7NZjn929oJ133JkPLrNEj=[e-mail]%3E), Spark is incompatible with Jersey 2 especially when Spark is embedded in an application running with Jersey.

There was an in-depth discussion on options for shading and making it easier for users to be able to use Jersey 2 with Spark applications: https://github.com/apache/spark/pull/9615

To recap the discussion, Jersey 1 has two issues:

1. It has classes listed in META-INF/services/ files that would be loaded even if Jersey 2 was being loaded on the classpath in a higher precedence. This means that Jersey 2 would attempt to use Jersey 1 implementations in some places regardless of user attempts to override the dependency with things like userClassPathFirst.
2. Jersey 1 packages javax.ws.rs classes inside itself, making it hard to exclude just javax.ws.rs APIs and replace them with ones that Jersey 2 is compatible with.

Also discussed was the fact that plain old shading doesn't work here, since you would need to shade lines in META-INF/services as well, not just classes. Not to mention that shading JAX-RS annotations is tricky as well.

To recap the discussion as what needs to happen Spark-side, we need to:

1. Create a "org.spark-project.jersey" artifact (loosely speaking) which is the Jersey 1 jar minus all the javax.ws.rs stuff (no need to actually shade/namespace the classes that way, just the artifact name)
2. Put all the javax.ws.rs stuff extracted from step 1 into its own artifact, say "org.spark-project.javax.ws.rs". (META-INF/services/javax.ws.rs* files live in this artifact as well)
3. Spark-core's pom depends on org.spark-project artifacts from step 1 and 2
4. Spark assembly excludes META-INF/services/javax.ws.rs.* - it turns out these files aren't actually necessary for Jersey 1 to function properly in general (we need to test this more however)

Now a user that wants to depend on Jersey 2, and is depending on Spark maven artifacts, would do the following in their application
1. Provide my own dependency on Jersey 2 and its transitive javax.ws.rs dependencies
2. In my application's dependencies, exclude org.spark-project.javax.ws.rs from spark-core. We keep org.spark-project.jersey because spark-core needs it, but it will use the javax.ws.rs classes that my application is providing.
3. Set spark.executor.userClassPathFirst=true and ship Jersey 2 and new javax.ws.rs jars to the executors

Issue Links

• relates to: SPARK-12154 Upgrade to Jersey 2
• links to: [Github] Pull Request #9615 (mccheah)