[ARROW-15410] [C++][Datasets] Improve memory usage of datasets API when scanning parquet

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/30892
• Assignee: Weston Pace
• Reporter: Weston Pace
• Created: 22/Jan/22 01:40
• Updated: 11/Jan/23 11:36
• Resolved: 22/Apr/22 22:54

Description

This is a more targeted fix to improve memory usage when scanning parquet files. It is related to broader issues like ARROW-14648 but those will likely take longer to fix. The goal here is to make it possible to scan large parquet datasets with many files where each file has reasonably sized row groups (e.g. 1 million rows). Currently we run out of memory scanning a configuration as simple as:

21 parquet files
Each parquet file has 10 million rows split into row groups of size 1 million

Issue Links

• is depended upon by: ARROW-15411 [C++][Datasets] Improve memory usage of datasets
• links to: GitHub Pull Request #12228