Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2015 Spark UI issues at scale
  3. SPARK-2017

web ui stage page becomes unresponsive when the number of tasks is large

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.4.2, 1.5.0
    • Component/s: Web UI
    • Labels:

      Description

      sc.parallelize(1 to 1000000, 1000000).count()
      

      The above code creates one million tasks to be executed. The stage detail web ui page takes forever to load (if it ever completes).

      There are again a few different alternatives:

      0. Limit the number of tasks we show.
      1. Pagination
      2. By default only show the aggregate metrics and failed tasks, and hide the successful ones.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                andrewor14 Andrew Or
                Reporter:
                rxin Reynold Xin
              • Votes:
                2 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: