Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23836

Support returning StructType to the level support in GroupedMap Arrow's "scalar" UDFS (or similar)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • PySpark
    • None

    Description

      Currently not all of the supported types can be returned from the scalar pandas UDF type. This means if someone wants to return a struct type doing a map operation right now they either have to do a "junk" groupBy or use the non-vectorized results.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-27240 Use pandas DataFrame for struct type argument in Scalar Pandas UDF.

          • Major - Major loss of function.
          • Resolved
          relates to

          Umbrella - An overarching type made of sub-tasks SPARK-21187 Complete support for remaining Spark data types in Arrow Converters

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #23900

          Activity

            People

              bryanc Bryan Cutler
              holden Holden Karau
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: