[OOZIE-3161] Spark job failing because of servlet-api-2.5 jar

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 4.3.1
• Assignee: Satish Saley
• Reporter: Satish Saley
• Created: 16/Jan/18 02:03
• Updated: 14/Feb/18 13:02
• Resolved: 17/Jan/18 01:43

Description

Spark Job fails with

18/01/15 15:08:38 ERROR SparkContext: Error initializing SparkContext.
java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package
	at java.lang.ClassLoader.checkCerts(ClassLoader.java:898)
	at java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.spark-project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:136)
	at org.spark-project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:129)
	at org.spark-project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:98)
	at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:126)
	at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:113)
	at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:78)
	at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:62)
	at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:62)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:62)
	at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:63)
	at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:76)
	at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:195)
	at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:146)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:473)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
	at org.apache.oozie.example.SparkFileCopy.main(SparkFileCopy.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)

spark-streaming-flume-sink_2.10 is fetching jetty:servlet-api-2.5:jar. We need to exclude this.
Here is dependency tree for sharelib/spark

[INFO] +- org.apache.spark:spark-streaming-flume_2.10:jar:1.6.1:compile
[INFO] | +- org.apache.spark:spark-streaming-flume-sink_2.10:jar:1.6.1:compile
[INFO] | | +- (org.apache.flume:flume-ng-sdk:jar:1.6.0:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flume:flume-ng-core:jar:1.6.0:compile - omitted for duplicate)
[INFO] | | +- (org.scala-lang:scala-library:jar:2.10.5:compile - omitted for duplicate)
[INFO] | | \- (org.spark-project.spark:unused:jar:1.0.0:compile - omitted for duplicate)
[INFO] | +- org.apache.flume:flume-ng-core:jar:1.6.0:compile
[INFO] | | +- (org.apache.flume:flume-ng-sdk:jar:1.6.0:compile - omitted for duplicate)
[INFO] | | +- org.apache.flume:flume-ng-configuration:jar:1.6.0:compile
[INFO] | | | +- (org.slf4j:slf4j-api:jar:1.6.6:compile - version managed from 1.6.1; omitted for duplicate)
[INFO] | | | +- (org.slf4j:slf4j-log4j12:jar:1.6.6:compile - version managed from 1.6.1; omitted for duplicate)
[INFO] | | | \- (org.apache.flume:flume-ng-sdk:jar:1.6.0:compile - omitted for duplicate)
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.6.6:compile - version managed from 1.6.1; omitted for duplicate)
[INFO] | | +- (commons-io:commons-io:jar:2.1:compile - version managed from 2.4; omitted for duplicate)
[INFO] | | +- (commons-codec:commons-codec:jar:1.4:compile - version managed from 1.8; omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:compile - version managed from 1.2.12; omitted for duplicate)
[INFO] | | +- (org.slf4j:slf4j-log4j12:jar:1.6.6:compile - version managed from 1.6.1; omitted for duplicate)
[INFO] | | +- (commons-cli:commons-cli:jar:1.2:compile - omitted for duplicate)
[INFO] | | +- (commons-lang:commons-lang:jar:2.4:compile - version managed from 2.5; omitted for duplicate)
[INFO] | | +- (org.apache.avro:avro:jar:1.7.4:compile - omitted for conflict with 1.7.7)
[INFO] | | +- (org.apache.avro:avro-ipc:jar:1.7.4:compile - omitted for conflict with 1.7.7)
[INFO] | | +- (joda-time:joda-time:jar:2.1:compile - version managed from 2.9; omitted for duplicate)
[INFO] | | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | | +- org.mortbay.jetty:jetty:jar:6.1.14:compile (version managed from 6.1.26)
[INFO] | | | +- (org.mortbay.jetty:jetty-util:jar:6.1.14:compile - omitted for conflict with 6.1.26)
[INFO] | | | \- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile
[INFO] | | +- com.google.code.gson:gson:jar:2.7:compile (version managed from 2.2.2)

To reproduce:
Build Oozie with spark-1 profile and deploy. Run the spark example (with yarn-master mode) which comes in Oozie examples.