[PARQUET-1698] [C++] Add reader option to pre-buffer entire serialized row group into memory

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: cpp-4.0.0
• Component/s: parquet-cpp
• Labels: pull-request-available
• Assignee: David Li
• Reporter: Wes McKinney
• Created: 21/Nov/19 04:15
• Updated: 24/Mar/21 16:09
• Resolved: 24/Mar/21 16:09

Description

In some scenarios (example: reading datasets from Amazon S3), reading columns independently and allowing unbridled Read calls to the underlying file handle can yield suboptimal performance. In such cases, it may be preferable to first read the entire serialized row group into memory then deserialize the constituent columns from this

Note that such an option would not be appropriate as a default behavior for all file handle types since low-selectivity reads (example: reading only 3 columns out of a file with 100 columns) will be suboptimal in some cases. I think it would be better for "high latency" file systems to opt into this option

cc fsaintjacques bkietz apitrou

Issue Links

• Dependent: PARQUET-1820 [C++] Use a column filter hint to inform read prefetching in Arrow reads
• duplicates: PARQUET-1820 [C++] Use a column filter hint to inform read prefetching in Arrow reads
• is related to: ARROW-11601 [C++][Dataset] Expose pre-buffering in ParquetFileFormatReaderOptions
• relates to: ARROW-8763 [C++] Create RandomAccessFile::WillNeed-like API
• relates to: ARROW-7995 [C++] IO: coalescing and caching read ranges
• links to: GitHub Pull Request #6138