[ARROW-15808] [Python] take function doesn't work when table has large row counts

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Python
• Environment: pyarrow 7.0.0, pyarrow 5.0
• External issue URL: https://github.com/apache/arrow/issues/31249
• Assignee: Unassigned
• Reporter: Zhe Wu
• Created: 28/Feb/22 20:26
• Updated: 11/Jan/23 11:39
• Resolved: 13/Apr/22 11:43

Description

Steps to reproduce the error:

 

import pyarrow as pa
a = pa.array(["123456789"]*260000000)
tb = pa.table([a], names=["a"])
tb.take([1])

 

ERROR message:
ArrowInvalid: offset overflow while concatenating arrays

Issue Links

• duplicates: ARROW-9773 [C++] Take kernel can't handle ChunkedArrays that don't fit in an Array