Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0, 2.4.0
    • Component/s: PySpark
    • Labels:
      None

      Description

      Currently, we don't support BinaryType in Pandas UDFs:

      >>> from pyspark.sql.functions import pandas_udf
      >>> pudf = pandas_udf(lambda x: x, "binary")
      >>> df = spark.createDataFrame([[bytearray("a")]])
      >>> df.select(pudf("_1")).show()
      ...
      TypeError: Unsupported type in conversion to Arrow: BinaryType
      

      Also, the grouped aggregate Pandas UDF fail fast on ArrayType but seems we can support this case.

      We should better clarify it in doc in Pandas UDFs, and fail fast with type checking ahead, rather than execution time.

      Please consider this case:

      pandas_udf(lambda x: x, BinaryType())  # we can fail fast at this stage because we know the schema ahead
      

        Attachments

          Activity

            People

            • Assignee:
              hyukjin.kwon Hyukjin Kwon
              Reporter:
              hyukjin.kwon Hyukjin Kwon
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: