[SPARK-39551] Add AQE invalid plan check

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.2.2, 3.3.1, 3.4.0
• Component/s: SQL
• Assignee: Wei Xue
• Reporter: Wei Xue
• Created: 22/Jun/22 02:49
• Updated: 02/Dec/22 02:23
• Resolved: 22/Jun/22 14:39

Description

AQE logical optimization rules can lead to invalid physical plans as certain physical plan nodes are not compatible with others. E.g., `BroadcastExchangeExec` can only work as a direct child of broadcast join nodes.

Logical optimizations, on the other hand, are not (and should not be) aware of such restrictions. So a general solution here is to check for invalid plans and throw exceptions, which can be caught by AQE replanning process. And if such an exception is captured, AQE can void the current replanning result and keep using the latest valid plan.

Issue Links

• fixes: SPARK-41336 BroadcastExchange does not support the execute() code path. when AQE enabled
• is duplicated by: SPARK-39447 Only non-broadcast query stage can propagate empty relation
• relates to: SPARK-39447 Only non-broadcast query stage can propagate empty relation
• links to: [Github] Pull Request #36953 (maryannxue)
• links to: [Github] Pull Request #36953 (maryannxue)
• links to: [Github] Pull Request #37108 (dongjoon-hyun)