Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31967

Loading jobs UI page takes 40 seconds

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.4.6, 3.0.1
    • Fix Version/s: 2.4.7, 3.0.1, 3.1.0
    • Component/s: Web UI
    • Labels:
      None

      Description

      In the latest master branch, I find that the job list page becomes very slow.
      To reproduce in local setup:

      spark.read.parquet("/tmp/p1").createOrReplaceTempView("t1")
      spark.read.parquet("/tmp/p2").createOrReplaceTempView("t2")
      (1 to 1000).map(_ =>  spark.sql("select * from t1, t2 where t1.value=t2.value").show())
      

      And that, open live UI: http://localhost:4040/
      The loading time is about 40 seconds.

      If we comment out the function call for `drawApplicationTimeline`, then the loading time is around 1 second.

        Attachments

        1. load_time.jpeg
          15 kB
          Gengliang Wang
        2. profile.png
          65 kB
          Gengliang Wang

          Activity

            People

            • Assignee:
              Gengliang.Wang Gengliang Wang
              Reporter:
              Gengliang.Wang Gengliang Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: