Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28345

PythonUDF predicate should be able to pushdown to join

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: PySpark, SQL
    • Labels:
      None

      Description

      A Filter predicate using PythonUDF can't be push down into join condition, currently. A predicate like that should be able to push down to join condition. For PythonUDFs that can't be evaluated in join condition, PullOutPythonUDFInJoinCondition will pull them out later.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                viirya L. C. Hsieh
                Reporter:
                viirya L. C. Hsieh
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: