[ARROW-2227] [Python] Table.from_pandas does not create chunked_arrays.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/1673
• Assignee: Wes McKinney
• Reporter: Left Screen
• Created: 27/Feb/18 16:02
• Updated: 11/Jan/23 07:19
• Resolved: 13/Mar/18 20:46

Description

When creating a large enough array, pyarrow raises an exception:

import numpy as np
import pandas as pd
import pyarrow as pa

x = list('1' * 2**31)
y = pd.DataFrame({'x': x})
t = pa.Table.from_pandas(y)
# ArrowInvalid: BinaryArrow cannot contain more than 2147483646 bytes, have 2147483647

The array should be chunked for the user. As is, data frames with >2 GiB in binary data will struggle to get into arrow.

Issue Links

• is related to: ARROW-3762 [C++] Parquet arrow::Table reads error when overflowing capacity of BinaryArray
• links to: GitHub Pull Request #1740