Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Currently, we are only supporting data pages whose headers are 64K or less (see parquet/column/serialized-page.cc. Since page headers can essentially be arbitrarily large (in pathological cases) because of the page statistics, if deserializing the page header fails, we should attempt to read a progressively larger amount of file data in effort to find the end of the page header.
As part of this (and to make testing easier!), the maximum data page header size should be configurable. We can write test cases by defining appropriate Statistics structs to yield serialized page headers of whatever desired size.
On malformed files, we may run past the end of the file, in such cases we should raise a reasonable exception.