Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27394

The staleness of UI may last minutes or hours when no tasks start or finish

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0, 2.4.1
    • Fix Version/s: 2.4.2, 3.0.0
    • Component/s: Web UI
    • Labels:
      None

      Description

      Run the following codes on a cluster that has at least 2 cores.

      sc.makeRDD(1 to 1000, 1000).foreach { i =>
        Thread.sleep(300000)
      }
      

      The jobs page will just show one running task.

      This is because when the second task event calls "AppStatusListener.maybeUpdate" for a job, it will just ignore since the gap between two events is smaller than `spark.ui.liveUpdate.period`.

      After the second task event, in the above case, because there won't be any other task events, the Spark UI will be always stale until the next task event gets fired (after 300 seconds).

        Attachments

          Activity

            People

            • Assignee:
              zsxwing Shixiong Zhu
              Reporter:
              zsxwing Shixiong Zhu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: