[SPARK-43803] Spark connect: improve handling of long running API - awaitTermination

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Connect, Structured Streaming
• Assignee: Peng Zhong
• Reporter: Peng Zhong
• Created: 25/May/23 22:10
• Updated: 08/Jun/23 23:59
• Resolved: 08/Jun/23 23:59

Description

Currently, long running APIs like awaitTermination() keep running on the server even if the client has disconnected. We need to improve that. The implementation should periodically check if the client side is disconnected and release the resources.