[SPARK-43973] Structured Streaming UI should display failed queries correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.2.0, 3.3.0, 3.4.0
• Fix Version/s: 3.4.1, 3.5.0
• Component/s: Web UI
• Assignee: Kris Mok
• Reporter: Kris Mok
• Created: 05/Jun/23 22:39
• Updated: 06/Jun/23 05:08
• Resolved: 06/Jun/23 05:08

Description

The Structured Streaming UI is designed to be able to show a query's status (active/finished/failed) and if failed, the error message.
Due to a bug in the implementation, the error message in QueryTerminatedEvent isn't being tracked by the UI data, so in turn the UI always shows failed queries as "finished".

Example:

implicit val ctx = spark.sqlContext
import org.apache.spark.sql.execution.streaming.MemoryStream

spark.conf.set("spark.sql.ansi.enabled", "true")

val inputData = MemoryStream[(Int, Int)]

val df = inputData.toDF().selectExpr("_1 / _2 as a")

inputData.addData((1, 2), (3, 4), (5, 6), (7, 0))
val testQuery = df.writeStream.format("memory").queryName("kristest").outputMode("append").start
testQuery.processAllAvailable()

Here we intentionally fail a query, but the Spark UI's Structured Streaming tab would show this as "FINISHED" without any errors, which is wrong.