[SPARK-35382] Fix lambda variable name issues in nested DataFrame functions in Python APIs

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.1.2, 3.2.0
• Component/s: PySpark
• Labels: correctness
• Assignee: Takuya Ueshin
• Reporter: Hyukjin Kwon
• Created: 12/May/21 04:11
• Updated: 12/Dec/22 18:10
• Resolved: 13/May/21 06:11

Description

Python side also has the same issue as SPARK-34794

from pyspark.sql.functions import *
df = sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters")
df.select(
    transform(
        "numbers",
        lambda n: transform("letters", lambda l: struct(n.alias("n"), l.alias("l")))
    )
).show()

+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|transform(numbers, lambdafunction(transform(letters, lambdafunction(struct(namedlambdavariable() AS n, namedlambdavariable() AS l), namedlambdavariable())), namedlambdavariable()))|
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                                                                [[{a, a}, {b, b},...|
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Issue Links

• relates to: SPARK-34794 Nested higher-order functions broken in DSL
• links to: [Github] Pull Request #32523 (ueshin)