Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41661

Support for User-defined Functions in Python

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Done
    • 3.4.0
    • None
    • Connect
    • None

    Description

      User-defined Functions in Python consist of (pickled) Python UDFs and (Arrow-optimized) Pandas UDFs. They enable users to run arbitrary Python code on top of the Apache Spark™ engine. Users only have to state "what to do"; PySpark, as a sandbox, encapsulates "how to do it".

      Spark Connect Python Client (SCPC), as a client and server interface for PySpark will eventually replace the legacy API of PySpark. Supporting PySpark UDFs is essential for Spark Connect to reach parity with the PySpark legacy API.

      See design doc here.

      Attachments

        Issue Links

          is depended upon by

          Umbrella - An overarching type made of sub-tasks SPARK-42393 Support for Pandas/Arrow Functions API

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue SPARK-42271 Reuse UDF test cases under `pyspark.sql.tests`

          • Major - Major loss of function.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              XinrongM Xinrong Meng
              grundprinzip-db Martin Grund
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: