Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-3850

[Python] Support MapType and StructType for enhanced PySpark integration

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.11.1
    • 2.0.0
    • Python
    • None

    Description

      It would be great to support MapType and (nested) StructType in Arrow so that PySpark can make use of it.

       
      Quite often as in my use-case in Hive table cells are also complex types saved. Currently it's not possible to user the new pandas_udf decorator which internally uses Arrow to generate a UDF for columns with complex types.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              FlorianWilhelm Florian Wilhelm
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: