[HADOOP-18750] Spark History Server 3.3.1 fails to starts with Hadoop 3.3.x

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: pull-request-available
• Target Version/s: 3.3.7
• Assignee: Kamal Sharma
• Reporter: Aman Raj
• Created: 24/May/23 04:20
• Updated: 29/Jun/23 18:49

Description

When Spark History Server tries to start with Hadoop 3.3.4 (Happens only in Kerberos scenarios), it fails to do so with the following exception: 

23/05/23 03:14:15 ERROR HistoryServer [main]: Failed to bind HistoryServer

java.lang.IllegalStateException: class org.apache.hadoop.security.authentication.server.AuthenticationFilter is not a javax.servlet.Filter

        at org.sparkproject.jetty.servlet.FilterHolder.doStart(FilterHolder.java:103) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.sparkproject.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:749) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_372]

        at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[?:1.8.0_372]

        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) ~[?:1.8.0_372]

        at org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:774) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:491) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.apache.spark.ui.WebUI.$anonfun$bind$3(WebUI.scala:154) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.apache.spark.ui.WebUI.$anonfun$bind$3$adapted(WebUI.scala:154) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) ~[scala-library-2.12.15.jar:?]

        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ~[scala-library-2.12.15.jar:?]

        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]

        at org.apache.spark.ui.WebUI.bind(WebUI.scala:154) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.apache.spark.deploy.history.HistoryServer.bind(HistoryServer.scala:164) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:310) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]

        at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala) ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT] 

 

Issue Links

• links to: GitHub Pull Request #5695