[SPARK-31967] Loading jobs UI page takes 40 seconds

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.6, 3.0.1
• Fix Version/s: 2.4.7, 3.0.1, 3.1.0
• Component/s: Web UI
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 11/Jun/20 07:58
• Updated: 14/Jun/20 19:21
• Resolved: 13/Jun/20 00:36

Description

In the latest master branch, I find that the job list page becomes very slow.
To reproduce in local setup:

spark.read.parquet("/tmp/p1").createOrReplaceTempView("t1")
spark.read.parquet("/tmp/p2").createOrReplaceTempView("t2")
(1 to 1000).map(_ =>  spark.sql("select * from t1, t2 where t1.value=t2.value").show())

And that, open live UI: http://localhost:4040/
The loading time is about 40 seconds.

If we comment out the function call for `drawApplicationTimeline`, then the loading time is around 1 second.

Issue Links

• relates to: SPARK-31971 Add pagination support for all jobs timeline
• links to: [Github] Pull Request #28806 (gengliangwang)
• links to: [Github] Pull Request #28806 (gengliangwang)
• links to: [Github] Pull Request #28811 (gengliangwang)
• links to: [Github] Pull Request #28811 (gengliangwang)
• links to: [Github] Pull Request #28813 (gengliangwang)