SPARK-37915 added CollapseProject in rule CombineUnions, but it shouldn't collapse projects that contain correlated subqueries since haven't been de-correlated (PullupCorrelatedPredicates).
Here is a simple example to reproduce this issue
SELECT (SELECT IF(x, 1, 0)) AS a FROM (SELECT true) t(x) UNION SELECT 1 AS a
java.lang.IllegalStateException: Couldn't find x#4 in 
- is caused by
SPARK-37915 Combine unions if there is a project between them
- links to