[ARROW-10172] [Python] pyarrow.concat_arrays segfaults if a resulting StringArray's capacity overflows

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.0.1, 2.0.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/26180
• Assignee: Unassigned
• Reporter: Artem KOZHEVNIKOV
• Created: 05/Oct/20 09:20
• Updated: 11/Jan/23 08:11
• Resolved: 23/Sep/22 15:10

Description

I'm sorry if this was already reported, but there's an overflow issue in concatenation of large arrays

In [1]: import pyarrow as pa

In [2]: str_array = pa.array(['a' * 128] * 10**8)

In [3]: large_array = pa.concat_arrays([str_array] * 50)
Segmentation fault (core dumped)

I suppose that this should be handled by upcast to large_string.

Issue Links

• is related to: ARROW-10799 [C++] Take on string chunked arrays slow and fails
• is superceded by: ARROW-17828 [C++][Python] Large strings cause ArrowInvalid: offset overflow while concatenating arrays
• relates to: ARROW-10494 .take silently overflow on list array (when casting to large_list is needed)
• relates to: ARROW-7731 [C++][Parquet] Support LargeListArray