Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21432

Reviving broken partial functions in UDF in PySpark

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: PySpark
    • Labels:
      None

      Description

      This is related with SPARK-21394

      We also happened to break partial function support in UDF.

      Spark 2.1:

      >>> from pyspark.sql import functions
      >>> from functools import partial
      >>>
      >>>
      >>> partial_func = partial(lambda x: x, x=1)
      >>> udf = functions.udf(partial_func)
      >>> spark.range(1).select(udf()).show()
      +---------+
      |partial()|
      +---------+
      |        1|
      +---------+
      

      master:

      >>> from pyspark.sql import functions
      >>> from functools import partial
      >>>
      >>>
      >>> partial_func = partial(lambda x: x, x=1)
      >>> udf = functions.udf(partial_func)
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File ".../spark/python/pyspark/sql/functions.py", line 2154, in udf
          return _udf(f=f, returnType=returnType)
        File ".../spark/python/pyspark/sql/functions.py", line 2145, in _udf
          return udf_obj._wrapped()
        File ".../spark/python/pyspark/sql/functions.py", line 2099, in _wrapped
          @functools.wraps(self.func, assigned=assignments)
        File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/functools.py", line 33, in update_wrapper
          setattr(wrapper, attr, getattr(wrapped, attr))
      AttributeError: 'functools.partial' object has no attribute '__module__'
      

        Attachments

          Activity

            People

            • Assignee:
              hyukjin.kwon Hyukjin Kwon
              Reporter:
              hyukjin.kwon Hyukjin Kwon
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: