[SPARK-2299] Consolidate various stageIdTo* hash maps

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Web UI
• Target Version/s: 1.1.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 26/Jun/14 21:04
• Updated: 18/Jul/14 01:59
• Resolved: 18/Jul/14 01:59

Description

In JobProgressListener:

  val stageIdToTime = HashMap[Int, Long]()
  val stageIdToShuffleRead = HashMap[Int, Long]()
  val stageIdToShuffleWrite = HashMap[Int, Long]()
  val stageIdToMemoryBytesSpilled = HashMap[Int, Long]()
  val stageIdToDiskBytesSpilled = HashMap[Int, Long]()
  val stageIdToTasksActive = HashMap[Int, HashMap[Long, TaskInfo]]()
  val stageIdToTasksComplete = HashMap[Int, Int]()
  val stageIdToTasksFailed = HashMap[Int, Int]()
  val stageIdToTaskData = HashMap[Int, HashMap[Long, TaskUIData]]()
  val stageIdToExecutorSummaries = HashMap[Int, HashMap[String, ExecutorSummary]]()
  val stageIdToPool = HashMap[Int, String]()
  val stageIdToDescription = HashMap[Int, String]()

We should consolidate them to reduce memory & be less error prone.

Issue Links

• is related to: SPARK-2298 Show stage attempt in UI
• links to: Github pull request