[SPARK-41441] Allow Generate with no required child output to host outer references

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Allison Wang
• Reporter: Allison Wang
• Created: 07/Dec/22 16:47
• Updated: 19/Dec/22 12:08
• Resolved: 19/Dec/22 12:08

Description

Currently, in CheckAnalysis, Spark disallows Generate to host any outer references when it's required child output is not empty. But when the child output is empty, it can host outer references, which DecorrelateInnerQuery does not handle.

For example,

select * from t, lateral (select explode(array(c1, c2)))

This throws an internal error :

Caused by: java.lang.AssertionError: assertion failed: Correlated column is not allowed in Generate explode(array(outer(c1#219), outer(c2#220))), false, [col#221] +- OneRowRelation

 We should support Generate to host outer references when its required child output is empty.

Issue Links

• links to: [Github] Pull Request #38968 (allisonwang-db)