Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27122

YARN test failures in Java 9+

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.4.5, 3.0.0
    • Component/s: Spark Core, YARN
    • Labels:
      None

      Description

      Currently on Java 11:

      YarnSchedulerBackendSuite:
      - RequestExecutors reflects node blacklist and is serializable
      - Respect user filters when adding AM IP filter *** FAILED ***
        java.lang.ClassCastException: org.spark_project.jetty.servlet.ServletContextHandler cannot be cast to org.eclipse.jetty.servlet.ServletContextHandler
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at scala.collection.TraversableLike.map(TraversableLike.scala:237)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2(YarnSchedulerBackend.scala:183)
        at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2$adapted(YarnSchedulerBackend.scala:174)
        at scala.Option.foreach(Option.scala:274)
        ...
      

      This looks like a classpath issue, probably ultimately related to the same classloader issues in https://issues.apache.org/jira/browse/SPARK-26839

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              ajithshetty Ajith S
              Reporter:
              srowen Sean R. Owen

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment