[SPARK-38550] Use a disk-based store to save more information in live UI to help debug

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core, SQL
• Assignee: Linhong Liu
• Reporter: Linhong Liu
• Created: 15/Mar/22 03:21
• Updated: 07/Nov/22 22:21
• Resolved: 14/Apr/22 06:01

Description

In Spark, the UI lacks troubleshooting abilities. For example:

• AQE plan changes are not available

• plan description of a large plan is truncated

This is because the live UI depends on an in-memory KV store. We should always be worried

about the stability issues when adding more information to the store.

Therefore, it's better to add a disk-based store to save more information

Issue Links

• links to: [Github] Pull Request #35856 (linhongliu-db)
• links to: [Github] Pull Request #35856 (linhongliu-db)
• links to: [Github] Pull Request #36274 (gengliangwang)
• links to: [Github] Pull Request #36274 (gengliangwang)
• links to: [Github] Pull Request #38542 (gengliangwang)