[SPARK-44913] DS V2 supports push down V2 UDF that has magic method

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Xianyang Liu
• Reporter: Xianyang Liu
• Created: 22/Aug/23 14:58
• Updated: 09/Oct/23 04:47
• Resolved: 29/Sep/23 23:37

Description

Right now we only support pushing down the V2 UDF that has not a magic method. Because the V2 UDF will be analyzed into the `ApplyFunctionExpression` which could be translated and pushed down. However, a V2 UDF that has the magic method will be analyzed into `StaticInvoke` or `Invoke` that can not be translated into V2 expression and then can not be pushed down to the data source. The magic method is suggested. So this PR adds the support of pushing down the V2 UDF that has a magic method.

Issue Links

• links to: [Github] Pull Request #42612 (ConeyLiu)
• links to: GitHub Pull Request #42612
• links to: GitHub Pull Request #43262