[SPARK-44432] Terminate streaming queries when a session times out in Spark Connect

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: Connect, Structured Streaming
• Epic Link: Streaming Spark Connect
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Created: 14/Jul/23 17:51
• Updated: 30/Jul/23 01:42
• Resolved: 30/Jul/23 01:42

Description

Spark Connect server keeps a session id at connect server alive as long as a streaming query is alive. This may not be desirable. If the client disconnects long enough and times out in server, we should let the session mapping to be removed and terminate the streaming queries.