[SPARK-21478] Unpersist a DF also unpersists related DFs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.1.1, 2.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Roberto Mirizzi
• Created: 19/Jul/17 23:11
• Updated: 25/May/21 08:13
• Resolved: 29/Jul/17 18:48

Description

Starting with Spark 2.1.1 I observed this bug. Here's are the steps to reproduce it:

1. create a DF
2. persist it
3. count the items in it
4. create a new DF as a transformation of the previous one
5. persist it
6. count the items in it
7. unpersist the first DF

Once you do that you will see that also the 2nd DF is gone.
The code to reproduce it is:

val x1 = Seq(1).toDF()
x1.persist()
x1.count()
assert(x1.storageLevel.useMemory)

val x11 = x1.select($"value" * 2)
x11.persist()
x11.count()
assert(x11.storageLevel.useMemory)

x1.unpersist()

assert(!x1.storageLevel.useMemory)
//the following assertion FAILS
assert(x11.storageLevel.useMemory)

Issue Links

• relates to: SPARK-24596 Non-cascading Cache Invalidation
• links to: [Github] Pull Request #18771 (vinodkc)