[SPARK-19504] clearCache fails to delete orphan RDDs, especially in pyspark

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.1.0
• Component/s: Optimizer, SQL
• Labels: bulk-closed execution optimizer plan query

Environment

Both pyspark and scala spark. Although scala spark uncaches some RDD types even if orphan

• Flags: Important
• Assignee: Unassigned
• Reporter: R
• Created: 07/Feb/17 22:41
• Updated: 17/May/20 17:58
• Resolved: 21/May/19 04:16

Description

x=sc.parallelize([1,3,10,9]).cache()
x.count()
x=sc.parallelize([1,3,10,9]).cache()
x.count()
sqlContex.clearCache()

Overwriting x will create an orphan RDD, which cannot be deleted with clearCache(). This happens in both scala and pyspark.

Similar thing happens for rdds created from dataframe in python
spark.read.csv(....).rdd()
However, in scala clearCache can get rid of some orphan rdd types.