[SPARK-34580] Provide the relationship between batch ID and SQL executions (and/or Jobs) in SS UI page

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Gabor Somogyi
• Created: 01/Mar/21 09:48
• Updated: 12/Dec/22 18:10

Description

The current SS UI page focuses to show the trends among the batches, which is great to figure out whether the streaming query is running healthy or not, and the oddness of specific batch.

One thing still bugging you is that what you can get from here is the batch ID (number), which means you have to find out related SQL executions and Jobs manually with the batch ID. It's high likely bound to the recent runs of SQL executions/Jobs so you may not need to find it with searching on lots of pages, but the fact you need to find it by yourself manually is still annoying.

It would be nice if we can provide the relationship between batch ID and SQL executions (probably Jobs as well if the space is enough) and links to these pages, like we see job page links from SQL execution page.