[SPARK-32954] Add jakarta.servlet-api test dependency to yarn module to avoid classpath badcase of UTs

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: YARN
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 21/Sep/20 14:45
• Updated: 24/Sep/20 15:33
• Resolved: 24/Sep/20 15:33

Description

When I tried to verify that the resource-managers yarn module passed all UTs in Scala 2.13 , I found that there are a classpath dependent badcase blocking the test because there are more than one `servlet-api` dependency in spark now:

• one is `javax.servlet:javax.servlet-api:3.10:compile` we config in core/pom.xml,
• the other is `jakarta.servlet:jakarta.servlet-api:4.0.3:test`  cascaded by `org.glassfish.jersey.test-framework.providers`, we can use `mvn dependency:tree` to check it .

So when we execute resource-managers yarn module test use

mvn clean test -pl resource-managers/yarn -Pyarn 

or 

mvn clean test -pl resource-managers/yarn -Pyarn -Pscala-2.13

 

if the position of `javax.servlet-api` in the  in classpath is before `jakarta.servlet-api`, there are some cases failed in YarnClusterSuite, YarnShuffleIntegrationSuite  and YarnShuffleAuthSuite.

 

The failed reason as follow:

 

20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: Exception in thread "main" java.lang.ExceptionInInitializerError
...
20/09/18 19:14:07.486 launcher-proc-1 INFO YarnClusterDriver: Caused by: java.lang.SecurityException: class "javax.servlet.http.HttpSessionIdListener"'s signer information does not match signer information of other classes in the same package
...

Issue Links

• links to: [Github] Pull Request #29824 (LuciferYang)