[SPARK-23030] Decrease memory consumption with toPandas() collection using Arrow

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: PySpark, SQL
• Assignee: Bryan Cutler
• Reporter: Bryan Cutler
• Created: 10/Jan/18 19:41
• Updated: 12/Dec/22 18:11
• Resolved: 29/Aug/18 07:02

Description

Currently with Arrow enabled, calling toPandas() results in a collection of all partitions in the JVM in the form of batches of Arrow file format. Once collected in the JVM, they are served to the Python driver process.

I believe using the Arrow stream format can help to optimize this and reduce memory consumption in the JVM by only loading one record batch at a time before sending it to Python. This might also reduce the latency between making the initial call in Python and receiving the first batch of records.

Issue Links

• is blocked by: SPARK-23874 Upgrade apache/arrow to 0.10.0
• relates to: SPARK-25274 Improve toPandas with Arrow by sending out-of-order record batches
• links to: [Github] Pull Request #21546 (BryanCutler)