[ARROW-4757] [C++] Nested chunked array support

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: C++
• External issue URL: https://github.com/apache/arrow/issues/21276
• Assignee: Unassigned
• Reporter: Philipp Moritz
• Created: 04/Mar/19 08:19
• Updated: 11/Jan/23 07:36
• Resolved: 21/Aug/19 23:31

Description

Dear all,

I'm currently trying to lift the 2GB limit on the python serialization. For this, I implemented a chunked union builder to split the array into smaller arrays.

However, some of the children of the union array can be ListArrays, which can themselves contain UnionArrays which can contain ListArrays etc. I'm at a bit of a loss how to handle this. In principle I'd like to chunk the children too. However, currently UnionArrays can only have children of type Array, and there is no way to treat a chunked array (which is a vector of Arrays) as an Array to store it as a child of a UnionArray. Any ideas how to best support this use case?

– Philipp.