[SPARK-11192] Spark sql seems to leak org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.5.1
• Component/s: SQL

Environment

java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
org.apache.spark/spark-sql_2.10 "1.5.1"

Embedded, in-process spark. Have not tested on standalone or yarn clusters.

• Assignee: Unassigned
• Reporter: Blake Livingston
• Created: 19/Oct/15 21:02
• Updated: 20/Oct/15 22:59
• Resolved: 20/Oct/15 22:59

Description

Noticed that slowly, over the course of a day or two, heap memory usage on a long running spark process increased monotonically.
After doing a heap dump and examining in jvisualvm, saw there were over 15M org.apache.spark.sql.execution.ui.SQLTaskMetrics objects allocated, taking over 500MB.

Issue Links

• duplicates: SPARK-11126 A memory leak in SQLListener._stageIdToStageMetrics