[SPARK-21579] dropTempView has a critical BUG

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 2.1.1, 2.2.0
• Component/s: SQL
• Flags: Important
• Assignee: Unassigned
• Reporter: ant_nebula
• Created: 31/Jul/17 03:29
• Updated: 19/Jun/18 21:32
• Resolved: 01/Aug/17 06:28

Description

when I dropTempView dwd_table1 only, sub table dwd_table2 also disappear from http://127.0.0.1:4040/storage/.
It affect version 2.1.1 and 2.2.0, 2.1.0 is ok for this problem.

val spark = SparkSession.builder.master("local").appName("sparkTest").getOrCreate()
val rows = Seq(Row("p1", 30), Row("p2", 20), Row("p3", 25), Row("p4", 10), Row("p5", 40), Row("p6", 15))
val schema = new StructType().add(StructField("name", StringType)).add(StructField("age", IntegerType))

val rowRDD = spark.sparkContext.parallelize(rows, 3)
val df = spark.createDataFrame(rowRDD, schema)
df.createOrReplaceTempView("ods_table")
spark.sql("cache table ods_table")

spark.sql("cache table dwd_table1 as select * from ods_table where age>=25")
spark.sql("cache table dwd_table2 as select * from dwd_table1 where name='p1'")
spark.catalog.dropTempView("dwd_table1")
//spark.catalog.dropTempView("ods_table")

spark.sql("select * from dwd_table2").show()

It will keep ods_table1 in memory, although it will not been used anymore. It waste memory, especially when my service diagram much more complex

Issue Links

• Blocked: SPARK-21607 Can dropTempView function add a param like dropTempView(viewName: String, dropSelfOnly: Boolean)
• relates to: SPARK-19765 UNCACHE TABLE should also un-cache all cached plans that refer to this table
• relates to: SPARK-24596 Non-cascading Cache Invalidation