[SPARK-49982] Negative AQE caching in SQL cache

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Ziqi Liu
• Reporter: Ziqi Liu
• Created: 15/Oct/24 23:29
• Updated: 24/Oct/24 02:30
• Resolved: 24/Oct/24 02:30

Description

When we use a cached an AQE plan, it will do `cachedPlan.execute` to build the RDD, which will execute all AQE stages except the result stage. If any of them failed, the failure will be cached by lazy RDD val. So the next time when we reuse that cached plan (even by a totally irrelevant caller) it will fail immediately.

We need to re-cache the AQE plan upon failure.

Issue Links

• links to: GitHub Pull Request #48486