[ARROW-9878] [Python] table to_pandas self_destruct=True + split_blocks=True cannot prevent doubling memory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.17.1, 1.0.0
• Fix Version/s: 4.0.0
• Component/s: Documentation, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25913
• Assignee: David Li
• Reporter: Weichen Xu
• Created: 28/Aug/20 03:41
• Updated: 11/Jan/23 08:09
• Resolved: 30/Mar/21 17:51

Description

Test on: pyarrow 1.0.1, system: Ubuntu 16.04, python3.7

 

Reproduce code:

Generate about 800MB data first.

import pyarrow as pa

# generate about 800MB data
data = [pa.array([10]* 1000)]
batch = pa.record_batch(data, names=['f0'])
with open('/tmp/t1.pa', 'wb') as f1:
	writer = pa.ipc.new_stream(f1, batch.schema)
	for i in range(100000):
		writer.write_batch(batch)
	writer.close()

Test to_pandas with self_destruct=True, split_blocks=True, use_threads=False

import pyarrow as pa
import time
import sys

import os
pid = os.getpid()
print(f'run `psrecord {pid} --plot /tmp/t001.png` and then press ENTER.')
sys.stdin.readline()

with open('/tmp/t1.pa', 'rb') as f1:
	reader = pa.ipc.open_stream(f1)
	batches = [b for b in reader]

pa_table = pa.Table.from_batches(batches)
del batches
time.sleep(3)
pdf = pa_table.to_pandas(self_destruct=True, split_blocks=True, use_threads=False)
del pa_table
time.sleep(3)

The attached file is psrecord profiling result.

Issue Links

• links to: GitHub Pull Request #9730