[SPARK-33435] DSv2: REFRESH TABLE should invalidate caches

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.0.2, 3.1.0
• Component/s: SQL
• Labels: DSv2 correctness
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 12/Nov/20 18:26
• Updated: 20/Nov/20 23:18
• Resolved: 12/Nov/20 23:23

Description

Currently, in DSv2 RefreshTableExec, we only invalidate metadata cache but not all the caches that referencing the table to be refreshed. This may cause correctness issue if these caches go stale and get queried later.

Note that since we don't support caching a v2 table yet, we can't recache the table itself at the moment.

Issue Links

• links to: [Github] Pull Request #30359 (sunchao)
• links to: [Github] Pull Request #30359 (sunchao)
• links to: [Github] Pull Request #30360 (sunchao)
• links to: [Github] Pull Request #30360 (sunchao)