[SPARK-34076] SQLContext.dropTempTable fails if cache is non-empty

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.1
• Component/s: SQL
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 11/Jan/21 22:05
• Updated: 13/Jan/21 13:22
• Resolved: 13/Jan/21 13:22

Description

SQLContext.dropTempView calls CatalogImpl.dropTempView which is implemented as following:

  override def dropTempView(viewName: String): Boolean = {
    sparkSession.sessionState.catalog.getTempView(viewName).exists { viewDef =>
      sparkSession.sharedState.cacheManager.uncacheQuery(
        sparkSession, viewDef, cascade = false)
      sessionCatalog.dropTempView(viewName)
    }
  }

Here, the logical plan viewDef is not resolved, and when passing to uncacheQuery, it could fail at sameResult call, when canonicalized plan is compared. The error message looks like:

Invalid call to qualifier on unresolved object, tree: 'key

This can be reproduced via:

  test("XXX") {
    val t = "t"
    val v = "v"
    withTable(t) {
      withTempView(v) {
        sql(s"CREATE TABLE $t AS SELECT * FROM src")
        sql(s"CACHE TABLE $t")
        sql(s"CREATE TEMPORARY VIEW $v AS SELECT key FROM src LIMIT 10")
      }
    }
  }

through test.

Issue Links

• is caused by: SPARK-33138 unify temp view and permanent view behaviors
• links to: [Github] Pull Request #31136 (sunchao)
• links to: [Github] Pull Request #31136 (sunchao)