[SPARK-24889] dataset.unpersist() doesn't update storage memory stats

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.2, 2.4.0
• Component/s: Spark Core
• Assignee: L. C. Hsieh
• Reporter: Yuri Bogomolov
• Created: 23/Jul/18 14:52
• Updated: 11/Sep/18 17:46
• Resolved: 11/Sep/18 17:46

Description

Steps to reproduce:

1) Start a Spark cluster, and check the storage memory value from the Spark Web UI "Executors" tab (it should be equal to zero if you just started)

2) Run:

val df = spark.sqlContext.range(1, 1000000000)
df.cache()
df.count()
df.unpersist(true)

3) Check the storage memory value again, now it's equal to 1GB

 

Looks like the memory is actually released, but stats aren't updated. This issue makes cluster management more complicated.

Issue Links

• is duplicated by: SPARK-25091 UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory
• links to: [Github] Pull Request #22341 (viirya)