[SPARK-48374] Support additional PyArrow Table column types

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.1
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Ian Cook
• Reporter: Ian Cook
• Created: 21/May/24 13:20
• Updated: 05/Jun/24 03:12
• Resolved: 03/Jun/24 00:23

Description

SPARK-48220 adds support for passing a PyArrow Table to createDataFrame(), but there are a few PyArrow column types that are not yet supported:

• fixed-size binary
• fixed-size list
• large list

 

Issue Links

• relates to: SPARK-48220 Allow passing PyArrow Table to createDataFrame()
• links to: GitHub Pull Request #46688
• links to: GitHub Pull Request #46872