[ARROW-12983] [C++][Python] Converter::Extend gets stuck in infinite loop causing OOM if values don't fit in single chunk

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 4.0.1
• Fix Version/s: 5.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/28701
• Assignee: David Li
• Reporter: Laurent Mazare
• Created: 05/Jun/21 12:37
• Updated: 11/Jan/23 08:30
• Resolved: 22/Jun/21 15:47

Description

Apologies if this is a duplicate, I haven't found anything related

When creating an arrow table via the python api, the following code runs out of memory after using all the available resources on a box with 512GB of ram. This happens with pyarrow 4.0.0 and 4.0.1. However when running the same code with pyarrow 3.0.0, the memory usage only reaches 5GB (which seems like the appropriate ballpark for the table size).
The code generates a table with a single string column with 1m rows, each string being 3000 characters long.

Not sure whether the issue is python related or not, I haven't tried replicating it from the C++ api.

 

import os, string
import numpy as np
import pyarrow as pa

print(pa.__version__)
np.random.seed(42)

alphabet = list(string.ascii_uppercase)

_col = []
for _n in range(1000):
  k = ''.join(np.random.choice(alphabet, 3000))
  _col += [k] * 1000

table = pa.Table.from_pydict({'col': _col})

Issue Links

• is duplicated by: ARROW-13014 [Python] Pandas to_feather no longer works - runs out of memory
• is duplicated by: ARROW-13254 [Python] Processes killed and semaphore objects leaked when reading pandas data
• is duplicated by: ARROW-13406 [Python] pyarrow.array memory leak on large string arrays
• links to: GitHub Pull Request #10470
• links to: GitHub Pull Request #10556