[ARROW-17828] [C++][Python] Large strings cause ArrowInvalid: offset overflow while concatenating arrays

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 9.0.0
• Component/s: C++, Python
• Labels: good-first-issue
• External issue URL: https://github.com/apache/arrow/issues/33049
• Assignee: Unassigned
• Reporter: Ben Epstein
• Created: 23/Sep/22 13:46
• Updated: 11/Jan/23 11:55

Description

When working with medium-sized datasets that have very long strings, arrow fails when trying to operate on the strings. The root is the `combine_chunks` function.

Here is a minimally reproducible example

import numpy as np
import pyarrow as pa

# Create a large string
x = str(np.random.randint(low=0,high=1000, size=(30000,)).tolist())
t = pa.chunked_array([x]*20_000)
# Combine the chunks into large string array - fails
combined = t.combine_chunks()

I get the following error

--------------------------------------------------------------------------- ArrowInvalid Traceback (most recent call last) /var/folders/x6/00594j4s2yv3swcn98bn8gxr0000gn/T/ipykernel_95780/4128956270.py in <module> ----> 1 z=t.combine_chunks()
~/.venv/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.ChunkedArray.combine_chunks() 
~/.venv/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays() ~/Documents/Github/dataquality/.venv/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status() ~.venv/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status() 
ArrowInvalid: offset overflow while concatenating arrays 

With smaller strings or smaller arrays this works fine.

x = str(np.random.randint(low=0,high=1000, size=(10,)).tolist())
t = pa.chunked_array([x]*1000)
combined = t.combine_chunks()

The first example that fails takes a few minutes to run. If you'd like a faster example for experimentation, you can use `vaex` to generate the chunked array much faster. This will throw the identical error and will run about 1 second.

import vaex
import numpy as np

n = 50_000
x = str(np.random.randint(low=0,high=1000, size=(30_000,)).tolist())
df = vaex.from_arrays(
    id=list(range(n)),
    y=np.random.randint(low=0,high=1000,size=n)
)
df["text"] = vaex.vconstant(x, len(df))
# text_chunk_array is now a pyarrow.lib.ChunkedArray
text_chunk_array = df.text.values
x = text_chunk_array.combine_chunks() 

 

 

 

Issue Links

• relates to: ARROW-13150 [Python] combine_chunks fails on column of table, but does not error on table itself
• relates to: ARROW-7245 [C++] Allow automatic String -> LargeString promotions when concatenating tables
• supercedes: ARROW-10172 [Python] pyarrow.concat_arrays segfaults if a resulting StringArray's capacity overflows