[SPARK-29087] Use DelegatingServletContextHandler to avoid CCE

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: DStreams, Spark Core
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 15/Sep/19 02:34
• Updated: 15/Sep/19 17:16
• Resolved: 15/Sep/19 17:16

Description

SPARK-27122 fixes `ClassCastException` at `yarn` module by using `DelegatingServletContextHandler`. Initially, this was discovered with JDK9+, but the class path issues affected in JDK8, too. This issue aims to fix `streaming` module.

$ build/mvn test -pl streaming
...
UISeleniumSuite:
- attaching and detaching a Streaming tab *** FAILED ***
  java.lang.ClassCastException: org.sparkproject.jetty.servlet.ServletContextHandler cannot be cast to org.eclipse.jetty.servlet.ServletContextHandler
...
Tests: succeeded 337, failed 1, canceled 0, ignored 1, pending 0
*** 1 TEST FAILED ***
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Issue Links

• is caused by: SPARK-28709 Memory leaks after stopping of StreamingContext
• relates to: SPARK-27122 YARN test failures in Java 9+
• links to: GitHub Pull Request #25791