[SPARK-26395] Spark Thrift server memory leak

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.2
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Konstantinos Andrikopoulos
• Created: 18/Dec/18 15:06
• Updated: 02/Mar/19 21:32
• Resolved: 01/Mar/19 23:21

Description

We are running Thrift Server in standalone mode and we have observed that the heap of the driver is constantly increasing. After analysing the heap dump the issue seems to be that the ElementTrackingStore is constantly increasing due to the addition of RDDOperationGraphWrapper objects that are not cleaned up.

The ElementTrackingStore defines the addTrigger method were you are able to set thresholds in order to perform cleanup but in practice it is used for  ExecutorSummaryWrapper, JobDataWrapper and StageDataWrapper classes by using the following spark properties

• spark.ui.retainedDeadExecutors
• spark.ui.retainedJobs
• spark.ui.retainedStages

So the  RDDOperationGraphWrapper which is been added using the onJobStart method of  AppStatusListener class kvstore.write(uigraph) #line 291

in not cleaned up and it constantly increases causing a memory leak

Issue Links

• duplicates: SPARK-25837 Web UI does not respect spark.ui.retainedJobs in some instances
• is duplicated by: SPARK-25814 spark driver runs out of memory on org.apache.spark.util.kvstore.InMemoryStore