[SPARK-42940] Session management support streaming connect

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Connect, Structured Streaming
• Epic Link: Streaming Spark Connect
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Created: 27/Mar/23 23:00
• Updated: 29/Apr/23 02:56
• Resolved: 29/Apr/23 02:56

Description

Add session support for streaming jobs. 

E.g. a session should stay alive when a streaming job is alive. 

It might differ more complex scenarios like what happens when client loses track of the session. Such semantics would be handled as part of session semantics across Spark Connect (including streaming). 

Issue Links

• links to: [Github] Pull Request #40937 (rangadi)