Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23836

Support returning StructType to the level support in GroupedMap Arrow's "scalar" UDFS (or similar)

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • PySpark
    • None

    Description

      Currently not all of the supported types can be returned from the scalar pandas UDF type. This means if someone wants to return a struct type doing a map operation right now they either have to do a "junk" groupBy or use the non-vectorized results.

      Attachments

        Issue Links

        is related to

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-27240 Use pandas DataFrame for struct type argument in Scalar Pandas UDF.

        • Major - Major loss of function.
        • Resolved
        relates to

        Umbrella - An overarching type made of sub-tasks SPARK-21187 Complete support for remaining Spark data types in Arrow Converters

        • Major - Major loss of function.
        • Resolved
        links to

        Web Link GitHub Pull Request #23900

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            bryanc Bryan Cutler Assign to me
            holden Holden Karau
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment