[ARROW-6281] [Python] Produce chunked arrays for nested types in pyarrow.array

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/22665
• Assignee: Krisztian Szucs
• Reporter: Wes McKinney
• Created: 16/Aug/19 18:44
• Updated: 11/Jan/23 07:46
• Resolved: 28/Sep/20 09:21

Description

As follow up to ARROW-5028 and other issues, in a case like

vals = [['x' * 1024]] * ((2 << 20) + 1)
arr = pa.array(vals)

The child array of the ListArray cannot hold all of the string data. After the patch for ARROW-5028, an exception is raised rather than returning a malformed array. We could (with some effort) instead produce a chunked array of list<string> type

Issue Links

• is fixed by: ARROW-9992 [C++][Python] Refactor python to arrow conversions based on a reusable conversion API
• is related to: ARROW-2367 [Python] ListArray has trouble with sizes greater than kMaximumCapacity