[SPARK-27337] QueryExecutionListener never cleans up listeners from the bus after SparkSession is cleared

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Vinoo Ganesh
• Created: 01/Apr/19 16:14
• Updated: 24/Jan/22 14:59

Description

As a result of https://github.com/apache/spark/commit/9690eba16efe6d25261934d8b73a221972b684f3, it looks like there is a memory leak (specifically https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala#L131). 

Because the Listener Bus on the context still has a reference to the listener (even after the SparkSession is cleared), they are never cleaned up. This means that if you close and remake spark sessions fairly frequently, you're leaking every single time.