[ARROW-14024] [C++] ScanOptions::batch_size not respected in parquet/IPC readers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.0.0
• Component/s: C++
• Labels: pull-request-available query-engine
• External issue URL: https://github.com/apache/arrow/issues/29622
• Assignee: David Li
• Reporter: Weston Pace
• Created: 17/Sep/21 11:50
• Updated: 11/Jan/23 08:37
• Resolved: 23/Sep/21 19:23

Description

At first glance it seems like Parquet's reader should work. The ScanOptions::batch_size property is forwarded into the ArrowReaderProperties for the parquet::arrow::FileReader. However, we then use ReadOneRowGroup which doesn't look at the batch_size option.

The IPC reader simply doesn't look at the property at all.

Even if we can't control the source read size (e.g. we have to read a full row group / record batch and have no control over its size) we can still split whatever we read into smaller batches that respect the batch size. This is important for achieving parallelism as we can then partition the CPU work across these batches.

Issue Links

• links to: GitHub Pull Request #11207