[SPARK-30927] StreamingQueryManager should avoid keeping reference to terminated StreamingQuery

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 23/Feb/20 21:49
• Updated: 12/Dec/22 18:10
• Resolved: 24/Feb/20 09:49

Description

Right now StreamingQueryManager will keep the last terminated query until "resetTerminated" is called. When the last terminated query has lots of states (a large sql plan, cached RDDs, etc.), it will waste these memory unnecessarily.

Issue Links

• links to: GitHub Pull Request #27678