Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
Skipping values in the go parquet column reader is effectively implemented by reading the target number of rows into scratch space which is then discarded. In the boolean case, BytesRequired returns returns a scratch buffer that allocates one bit per row, however that same scratch space is also attempted to be used for `defLvls` and `repLvls` (both int16), which requires two bytes per row. Since the boolean `values` buffer is not large enough to hold the same number of rows worth of def and rep levels, skipping too many rows results in an index out of bounds panic.
Note that for other column types, this does not seem to be an issue since the buffer needed for `values` is always larger than the buffer needed for def and rep levels, however there still seems to be no reason to include any non-nil value to `cr.ReadBatch(...)` for rep and def lvls when skipping any column in the reader.
