Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21478

Unpersist a DF also unpersists related DFs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 2.1.1, 2.2.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      Starting with Spark 2.1.1 I observed this bug. Here's are the steps to reproduce it:

      1. create a DF
      2. persist it
      3. count the items in it
      4. create a new DF as a transformation of the previous one
      5. persist it
      6. count the items in it
      7. unpersist the first DF

      Once you do that you will see that also the 2nd DF is gone.
      The code to reproduce it is:

      val x1 = Seq(1).toDF()
      x1.persist()
      x1.count()
      assert(x1.storageLevel.useMemory)
      
      val x11 = x1.select($"value" * 2)
      x11.persist()
      x11.count()
      assert(x11.storageLevel.useMemory)
      
      x1.unpersist()
      
      assert(!x1.storageLevel.useMemory)
      //the following assertion FAILS
      assert(x11.storageLevel.useMemory)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                roberto.mirizzi Roberto Mirizzi
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: