Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6553

Support for functools.partial as UserDefinedFunction

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.1, 1.4.0
    • Component/s: PySpark
    • Labels:

      Description

      Currently functools.partial s cannot be used as UserDefinedFunction s for DataFrame s, as the __name__ attribute does not exist. Passing a functools.partial object will raise an Exception at https://github.com/apache/spark/blob/master/python/pyspark/sql/functions.py#L126.

      functools.partial is very widely used and should probably be supported, despite its lack of a __name__.

      My suggestion is to use f.__repr__() instead, or check with hasattr(f, '__name__) and use __class__ if False.

        Attachments

          Activity

            People

            • Assignee:
              kalle Kalle Jepsen
              Reporter:
              kalle Kalle Jepsen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: