[SPARK-2015] Spark UI issues at scale

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.0.0
• Component/s: Web UI
• Epic Link: Monitoring and debugging improvements (Spark 1.2)
• Assignee: Unassigned
• Reporter: Reynold Xin
• Created: 04/Jun/14 08:27
• Updated: 10/Sep/15 14:47
• Resolved: 10/Sep/15 14:47

Description

This is an umbrella ticket for issues related to Spark's web ui when we run Spark at scale (large datasets, large number of machines, or large number of tasks).

Issue Links

• is duplicated by: SPARK-8667 Improve Spark UI behavior at scale
• relates to: SPARK-2501 Handle stage re-submissions properly in the UI
• relates to: SPARK-2147 Master UI forgets about Executors when application exits cleanly
• relates to: SPARK-2267 Log exception when TaskResultGetter fails to fetch/deserialize task result
• relates to: SPARK-2284 Failed tasks reported as success if the failure reason is not ExceptionFailure
• relates to: SPARK-2319 Number of tasks on executors become negative after executor failures
• relates to: SPARK-2298 Show stage attempt in UI
• relates to: SPARK-2286 Report exception/errors for failed tasks that are not ExceptionFailure
• relates to: SPARK-2297 Make task attempt and speculation more explicit in UI

Sub-Tasks

• rdd in-memory storage UI becomes unresponsive when the number of RDD partitions is large: Resolved
• web ui stage page becomes unresponsive when the number of tasks is large: Closed
• web ui should not remove executors if they are dead: Resolved
• Enable GZip for Web UI: Resolved
• Defer executing drawTaskAssignmentTimeline until page loaded to avoid to freeze the page: Resolved
• Avoid massive concating strings in Javascript: Resolved
• SparkUI stage page hangs with many tasks: Resolved
• Paginate stage page to avoid OOM with > 100,000 tasks: Closed