[SPARK-11126] A memory leak in SQLListener._stageIdToStageMetrics

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.5.1
• Fix Version/s: 1.5.2, 1.6.0
• Component/s: SQL
• Target Version/s: 1.5.2, 1.6.0
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 15/Oct/15 06:26
• Updated: 21/Jan/16 10:05
• Resolved: 18/Oct/15 20:52

Description

SQLListener adds all stage infos to _stageIdToStageMetrics, but only removes stage infos belonging to SQL executions.

Reported by Terry Hoo in https://www.mail-archive.com/[e-mail]/msg38810.html

Issue Links

• is duplicated by: SPARK-12906 LongSQLMetricValue cause memory leak on Spark 1.5.1
• is duplicated by: SPARK-11192 Spark sql seems to leak org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time
• links to: [Github] Pull Request #9132 (zsxwing)
• links to: [Github] Pull Request #9163 (zsxwing)