[SPARK-35636] Do not push down extract value in higher order function that references both sides of a join

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Karen Feng
• Reporter: Karen Feng
• Created: 03/Jun/21 18:04
• Updated: 12/Dec/22 18:10
• Resolved: 04/Jun/21 06:50

Description

Currently, lambda keys can be referenced outside of the lambda function:

Project transform(keys#0, lambdafunction(_extract_v1#0, lambda key#0, false)) AS a#0
+- 'Join Cross
:- Project kvs#0[lambda key#0].v1 AS _extract_v1#0
: +- LocalRelation <empty>, kvs#0
+- LocalRelation <empty>, keys#0

This should be unchanged from the original state:

Project transform(keys#418, lambdafunction(kvs#417[lambda key#420].v1, lambda key#420, false)) AS a#419
+- Join Cross
:- LocalRelation <empty>, kvs#417
+- LocalRelation <empty>, keys#418

Issue Links

• links to: [Github] Pull Request #32773 (karenfeng)
• links to: [Github] Pull Request #32773 (karenfeng)
• links to: [Github] Pull Request #32812 (HyukjinKwon)