[SPARK-35381] Fix lambda variable name issues in nested DataFrame functions in R APIs

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.1.2, 3.2.0
• Component/s: SparkR
• Labels: correctness
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 12/May/21 04:03
• Updated: 12/Dec/22 18:10
• Resolved: 12/May/21 07:54

Description

R's higher order functions also have the same problem with SPARK-34794:

df <- sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters")
collect(select(
  df,
  array_transform("numbers", function(number) {
    array_transform("letters", function(latter) {
      struct(alias(number, "n"), alias(latter, "l"))
    })
  })
))

  transform(numbers, lambdafunction(transform(letters, lambdafunction(struct(namedlambdavariable() AS n, namedlambdavariable() AS l), namedlambdavariable())), namedlambdavariable()))
1                                                                                                                                 a, a, b, b, c, c, a, a, b, b, c, c, a, a, b, b, c, c

Issue Links

• relates to: SPARK-34794 Nested higher-order functions broken in DSL
• links to: [Github] Pull Request #32517 (HyukjinKwon)
• links to: [Github] Pull Request #32517 (HyukjinKwon)