[SPARK-21187] Complete support for remaining Spark data types in Arrow Converters

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 3.1.0
• Component/s: PySpark, SQL
• Assignee: Bryan Cutler
• Reporter: Bryan Cutler
• Created: 23/Jun/17 05:11
• Updated: 12/Dec/22 18:11
• Resolved: 20/Nov/20 19:52

Description

This is to track adding the remaining type support in Arrow Converters. Currently, only primitive data types are supported. '

Remaining types:

• Date
• Timestamp
• Complex: Struct, Array, Map
• Decimal
• Binary
• Categorical when converting from Pandas

Some things to do before closing this out:

• Look to upgrading to Arrow 0.7 for better Decimal support (can now write values as BigDecimal)
• Need to add some user docs
• Make sure Python tests are thorough
• Check into complex type support mentioned in comments by leif, should we support mulit-indexing?

Issue Links

• is part of: SPARK-22216 Improving PySpark/Pandas interoperability
• is related to: SPARK-23836 Support returning StructType to the level support in GroupedMap Arrow's "scalar" UDFS (or similar)
• is related to: SPARK-27834 Make separate PySpark/SparkR vectorization configurations
• is related to: SPARK-26759 Arrow optimization in SparkR's interoperability
• relates to: SPARK-32285 Add PySpark support for nested timestamps with arrow
• relates to: SPARK-33489 Support null for conversion from and to Arrow type