[SPARK-39216] Do not collapse projects in CombineUnions if it hasCorrelatedSubquery

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Yuming Wang
• Reporter: Allison Wang
• Created: 18/May/22 02:15
• Updated: 24/May/22 17:10
• Resolved: 19/May/22 06:38

Description

 

SPARK-37915 added CollapseProject in rule CombineUnions, but it shouldn't collapse projects that contain correlated subqueries since haven't been de-correlated (PullupCorrelatedPredicates).

Here is a simple example to reproduce this issue

SELECT (SELECT IF(x, 1, 0)) AS a
FROM (SELECT true) t(x)
UNION 
SELECT 1 AS a 

Exception:

java.lang.IllegalStateException: Couldn't find x#4 in [] 

 

Issue Links

• is caused by: SPARK-37915 Combine unions if there is a project between them
• links to: [Github] Pull Request #36595 (wangyum)
• links to: [Github] Pull Request #36595 (wangyum)