[SPARK-2017] web ui stage page becomes unresponsive when the number of tasks is large

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.4.2, 1.5.0
• Component/s: Web UI
• Labels: starter
• Target Version/s: 1.4.2, 1.5.0
• Assignee: Andrew Or
• Reporter: Reynold Xin
• Created: 04/Jun/14 08:34
• Updated: 07/Aug/15 21:36
• Resolved: 09/Jul/15 20:24

Description

sc.parallelize(1 to 1000000, 1000000).count()

The above code creates one million tasks to be executed. The stage detail web ui page takes forever to load (if it ever completes).

There are again a few different alternatives:

0. Limit the number of tasks we show.
1. Pagination
2. By default only show the aggregate metrics and failed tasks, and hide the successful ones.

Issue Links

• is duplicated by: SPARK-7716 SparkUI stage page hangs with many tasks
• is related to: SPARK-4598 Paginate stage page to avoid OOM with > 100,000 tasks
• links to: [Github] Pull Request #1682 (carlosfuertes)
• links to: [Github] Pull Request #7296 (andrewor14)