[SPARK-47365] Add toArrow() DataFrame method to PySpark

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.1
• Fix Version/s: 4.0.0
• Component/s: Connect, Input/Output, PySpark, SQL
• Labels: pull-request-available

Language

Python

• Assignee: Ian Cook
• Reporter: Ian Cook
• Created: 12/Mar/24 18:59
• Updated: 09/May/24 13:43
• Resolved: 09/May/24 08:25

Description

Over in the Apache Arrow community, we hear from a lot of users who want to return the contents of a PySpark DataFrame as a PyArrow Table. Currently the only documented way to do this is:

PySpark DataFrame --> pandas DataFrame --> PyArrow Table

This adds significant overhead compared to going direct from PySpark DataFrame to PyArrow Table. Since PySpark already goes through PyArrow to convert to pandas, would it be possible to publicly expose a toArrow() method of the Spark DataFrame class?

Issue Links

• contains: SPARK-47465 Remove experimental tag from toArrow() PySpark DataFrame method
• is related to: SPARK-48220 Allow passing PyArrow Table to createDataFrame()
• is related to: SPARK-47466 Add PySpark DataFrame method to return iterator of PyArrow RecordBatches
• links to: GitHub Pull Request #45481