[SPARK-31354] SparkSession Lifecycle methods to fix memory leak

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Vinoo Ganesh
• Reporter: Vinoo Ganesh
• Created: 05/Apr/20 14:36
• Updated: 03/Jul/20 03:32
• Resolved: 21/May/20 16:07

Description

Follow up to https://issues.apache.org/jira/browse/SPARK-27958 after discussion on https://github.com/apache/spark/pull/24807. 

 

Let's instead expose methods that allow the user to manually clean up (terminate) a SparkSession, that also remove the listenerState from the context. 

Issue Links

• relates to: SPARK-32165 SessionState leaks SparkListener with multiple SparkSession
• links to: GitHub Pull Request #28128