[ARROW-14026] [C++] Batch readahead not working correctly in Parquet scanner

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.0.0
• Component/s: C++
• Labels: pull-request-available query-engine
• External issue URL: https://github.com/apache/arrow/issues/29624
• Assignee: David Li
• Reporter: Weston Pace
• Created: 17/Sep/21 11:59
• Updated: 11/Jan/23 08:37
• Resolved: 27/Sep/21 17:46

Description

The parquet scanner implements batch readahead by applying a readahead generator to the generator returned by parquet::arrow::FileReader::GetRecordBatchGenerator. However, that generator is constructed with MakeConcatenatedGenerator which, regrettably, has this comment:

> This generator is async-reentrant but will never pull from source reentrantly and will never pull from any subscription reentrantly.

This effectively prevents any batch readahead from happening and the file is always read one batch at a time. Part of the problem seems to be that ReadOneRowGroup in reader.cc returns a RecordBatchGenerator when it seems it should be able to return a RecordBatch. For the testing I am doing I changed this to return a single record batch which allowed me to get rid of the concatenated generator and batch readahead appeared to work properly but I didn't fully confirm the correctness of this.

Issue Links

• links to: GitHub Pull Request #11189