[SPARK-33311] Improve semantics for REFRESH TABLE

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Chao Sun
• Created: 31/Oct/20 19:11
• Updated: 14/Jan/21 03:01

Description

Currently, the semantics for REFRESH TABLE t is not well defined for view (let's say view) that reference the table t:

1. If view is cached, the behavior is not well-defined. Should Spark invalidate the cache (current behavior) or recache it?
2. If view is a temporary view, currently refreshing t does not refresh view since it will just reuse the logical plan defined in the session catalog. This could lead query failures (although with a helpful error message) or to incorrect results depending on the refresh behavior.

I think we should clear define and document the behavior here, so that users won't get confused.

Issue Links

• is related to: SPARK-33290 REFRESH TABLE should invalidate cache even though the table itself may not be cached
• relates to: SPARK-34052 A cached view should become invalid after a table is dropped