sc.parallelize(1 to 1000000, 1000000).count()
The above code creates one million tasks to be executed. The stage detail web ui page takes forever to load (if it ever completes).
There are again a few different alternatives:
0. Limit the number of tasks we show.
1. Pagination
2. By default only show the aggregate metrics and failed tasks, and hide the successful ones.
- is duplicated by
-
SPARK-7716 SparkUI stage page hangs with many tasks
-
- Resolved
-
- is related to
-
SPARK-4598 Paginate stage page to avoid OOM with > 100,000 tasks
-
- Closed
-
- links to