[SPARK-21187] Complete support for remaining Spark data types in Arrow Converters

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 3.1.0
• Component/s: PySpark, SQL
• Assignee: Bryan Cutler
• Reporter: Bryan Cutler
• Created: 23/Jun/17 05:11
• Updated: 12/Dec/22 18:11
• Resolved: 20/Nov/20 19:52

Description

This is to track adding the remaining type support in Arrow Converters. Currently, only primitive data types are supported. '

Remaining types:

• Date
• Timestamp
• Complex: Struct, Array, Map
• Decimal
• Binary
• Categorical when converting from Pandas

Some things to do before closing this out:

• Look to upgrading to Arrow 0.7 for better Decimal support (can now write values as BigDecimal)
• Need to add some user docs
• Make sure Python tests are thorough
• Check into complex type support mentioned in comments by Leif Mortenson, should we support mulit-indexing?

Issue Links

• is part of: SPARK-22216 Improving PySpark/Pandas interoperability
• is related to: SPARK-23836 Support returning StructType to the level support in GroupedMap Arrow's "scalar" UDFS (or similar)
• is related to: SPARK-27834 Make separate PySpark/SparkR vectorization configurations
• is related to: SPARK-26759 Arrow optimization in SparkR's interoperability
• relates to: SPARK-32285 Add PySpark support for nested timestamps with arrow
• relates to: SPARK-33489 Support null for conversion from and to Arrow type

Sub-Tasks

• Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas: Resolved
• Refactor ArrowConverters and add ArrayType and StructType support.: Resolved
• Add decimal type support to ArrowWriter.: Resolved
• Add User Documentation for Working with Arrow in Spark: Resolved
• Add ArrayType Support for working with Pandas and Arrow: Resolved
• Add date and timestamp support to ArrowConverters for toPandas() collection: Resolved
• Add BinaryType support for Arrow in PySpark: Resolved
• Add MapType support for Arrow Java: Resolved
• Need to apply pyarrow adjustments to complex types with DateType/TimestampType: Resolved
• Add MapType Support for Arrow in PySpark: Resolved
• Handle Pandas category type when converting from Python with Arrow: Resolved
• Document the features that require Pyarrow 0.10: Resolved