Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.0.0
Description
When computing definition levels for deeply nested arrays that include lists, the definition levels are correctly calculated, but they are not translated into correct indexes for the eventual primitive arrays.
For example, an int32 array could have no null values, but be a child of a list that has null values. If say the first 5 values of the int32 array are members of the first list item (i.e. list_array[0] = [1,2,3,4,5], and that list is itself a child of a struct whose index is null, the whole 5 values of the int32 array should be skipped. Further, the list's definition and repetition levels will be represented by 1 slot instead of the 5.
The current logic cannot cater for this, and potentially results in slicing the int32 array incorrectly (sometimes including some of those first 5 values).
This Jira is for the work necessary to compute the index into the eventual leaf arrays correctly.
I started doing it as part of the initial writer PR, but it's complex and is blocking progress.
Attachments
Issue Links
- links to
