[SPARK-28345] PythonUDF predicate should be able to pushdown to join

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: PySpark, SQL
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 11/Jul/19 04:54
• Updated: 12/Dec/22 18:10
• Resolved: 16/Jul/19 07:16

Description

A Filter predicate using PythonUDF can't be push down into join condition, currently. A predicate like that should be able to push down to join condition. For PythonUDFs that can't be evaluated in join condition, PullOutPythonUDFInJoinCondition will pull them out later.

Issue Links

• is related to: SPARK-28276 Convert and port 'cross-join.sql' into UDF test base
• links to: GitHub Pull Request #25106