[SPARK-17672] Spark 2.0 history server web Ui takes too long for a single application

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.1
• Component/s: Web UI
• Target Version/s: 2.0.1
• Assignee: Gang Wu
• Reporter: Gang Wu
• Created: 26/Sep/16 22:53
• Updated: 29/Sep/16 19:49
• Resolved: 29/Sep/16 19:48

Description

When there are 10K application history in the history server back end, it can take a very long time to even get a single application history page. After some investigation, I found the root cause was the following piece of code:

OneApplicationResource.scala

@Produces(Array(MediaType.APPLICATION_JSON))
private[v1] class OneApplicationResource(uiRoot: UIRoot) {

  @GET
  def getApp(@PathParam("appId") appId: String): ApplicationInfo = {
    val apps = uiRoot.getApplicationInfoList.find { _.id == appId }
    apps.getOrElse(throw new NotFoundException("unknown app: " + appId))
  }

}

Although all application history infos are stored in a LinkedHashMap, here to code transforms the map to an iterator and then uses the find() api which is O( n) instead of O(1) from a map.get() operation.

Issue Links

• links to: [Github] Pull Request #15247 (wgtmac)