Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12784

Spark UI IndexOutOfBoundsException with dynamic allocation

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.2, 1.6.0
    • Fix Version/s: 1.5.3, 1.6.1, 2.0.0
    • Component/s: Web UI, YARN
    • Labels:
      None

      Description

      Trying to load the web UI Executors page when using dynamic allocation running on yarn can lead to an IndexOutOfBoundsException Exception.

      I'm assuming the number of executors is changing as its trying to be loaded which is causing this as during this time it was letting executors go.

      HTTP ERROR 500

      Problem accessing /executors/. Reason:

      Server Error

      Caused by:

      java.lang.IndexOutOfBoundsException: 1058
      at scala.collection.LinearSeqOptimized$class.apply(LinearSeqOptimized.scala:52)
      at scala.collection.immutable.Stream.apply(Stream.scala:185)
      at org.apache.spark.ui.exec.ExecutorsPage$.getExecInfo(ExecutorsPage.scala:180)
      at org.apache.spark.ui.exec.ExecutorsPage$$anonfun$11.apply(ExecutorsPage.scala:60)
      at org.apache.spark.ui.exec.ExecutorsPage$$anonfun$11.apply(ExecutorsPage.scala:59)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      at scala.collection.immutable.Range.foreach(Range.scala:141)
      at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
      at scala.collection.AbstractTraversable.map(Traversable.scala:105)
      at org.apache.spark.ui.exec.ExecutorsPage.render(ExecutorsPage.scala:59)
      at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:79)
      at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:79)
      at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:69)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      at org.spark-project.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
      at org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
      at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:164)
      at org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      at org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      at org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      at org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      at org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      at org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      at org.spark-project.jetty.server.handler.GzipHandler.handle(GzipHandler.java:264)
      at org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
      at org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      at org.spark-project.jetty.server.Server.handle(Server.java:370)
      at org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
      at org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
      at org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
      at org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644)
      at org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
      at org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      at org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
      at org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      at java.lang.Thread.run(Thread.java:745)

      Simply reloading eventually gets the ui to come up so its not a blocker but not a very friendly experience either.

        Attachments

          Activity

            People

            • Assignee:
              zsxwing Shixiong Zhu
              Reporter:
              tgraves Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: