Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2015 Spark UI issues at scale
  3. SPARK-2016

rdd in-memory storage UI becomes unresponsive when the number of RDD partitions is large

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.0
    • 1.5.0
    • Web UI

    Description

      Try run

      sc.parallelize(1 to 100, 1000000).cache().count()
      

      And open the storage UI for this RDD. It takes forever to load the page.

      When the number of partitions is very large, I think there are a few alternatives:

      0. Only show the top 1000.
      1. Pagination
      2. Instead of grouping by RDD blocks, group by executors

      Attachments

        Activity

          People

            carsonwang Carson Wang
            rxin Reynold Xin
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: