[SPARK-34087] a memory leak occurs when we clone the spark session

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: wuyi
• Reporter: Fu Chen
• Created: 12/Jan/21 11:06
• Updated: 12/Apr/21 00:57
• Resolved: 23/Mar/21 07:39

Description

In Spark-3.0.1, the memory leak occurs when we keep cloning the spark session because a new ExecutionListenerBus instance will add to AsyncEventQueue when we clone a new session.

Issue Links

• is duplicated by: SPARK-34689 Spark Thrift Server: Memory leak for SparkSession objects
• links to: [Github] Pull Request #31156 (cfmcgrady)
• links to: [Github] Pull Request #31156 (cfmcgrady)
• links to: [Github] Pull Request #31231 (cfmcgrady)
• links to: [Github] Pull Request #31839 (Ngone51)
• links to: [Github] Pull Request #31881 (Ngone51)
• links to: [Github] Pull Request #31919 (Ngone51)
• links to: [Github] Pull Request #31919 (Ngone51)