[ARROW-9994] [C++][Python] Auto chunking nested array containing binary-like fields result malformed output

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 2.0.0
• Component/s: C++, Python
• External issue URL: https://github.com/apache/arrow/issues/26019
• Assignee: Krisztian Szucs
• Reporter: Krisztian Szucs
• Created: 14/Sep/20 15:20
• Updated: 11/Jan/23 08:10
• Resolved: 28/Sep/20 09:23

Description

In case of nested types the binary-like arrays are chunked but not the others, so after finalizing the builder the nested output array contains different length children.

   char = b'x'
   ty = pa.binary()

    v1 = char * 100000000
    v2 = char * 147483646

    struct_type = pa.struct([
        pa.field('bool', pa.bool_()),
        pa.field('integer', pa.int64()),
        pa.field('string-like', ty),
    ])

    data = [{'bool': True, 'integer': 1, 'string-like': v1}] * 20
    data.append({'bool': True, 'integer': 1, 'string-like': v2})
    arr = pa.array(data, type=struct_type)
    assert isinstance(arr, pa.Array)

    data.append({'bool': True, 'integer': 1, 'string-like': char})
    arr = pa.array(data, type=struct_type)
    assert isinstance(arr, pa.ChunkedArray)

len(arr.field(0)) == 22
len(arr.field(1)) == 22
len(arr.field(2)) == 1  # the string array gets chunked whereas the rest of the fields do not

Issue Links

• is fixed by: ARROW-9992 [C++][Python] Refactor python to arrow conversions based on a reusable conversion API