[SPARK-32076] Structured Streaming application continuity when encountering streaming query task level error

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.5
• Component/s: Structured Streaming

Environment

Apache Spark 2.4.5 Scala 2.11 without Hadoop

Hadoop 2.7.3 - YARN cluster

delta-core_ 2.11:0.6.1

• Assignee: Unassigned
• Reporter: Jim Huang
• Created: 23/Jun/20 14:26
• Updated: 23/Jun/20 14:26

Description

From the Spark Structured Streaming application continuity perspective, the thread that ran this task was terminated with ERROR SPARK-31995 but to YARN it is still an active running job even though this instance of the Spark Structured Streaming job is no longer making any further processing.  If the monitoring of the Spark Structured Streaming job is done only from the YARN job perspective, it may provide a false status.  In this situation, should the Spark Structure Streaming application fail hard and completely (fail by Spark framework or Application exception handling)?  Or should the developer investigate and develop some ideal monitoring implementation that has the right level of specificity to detect Spark Structured Streaming task level failures?  Any references on these topics are much appreciated.