[ARROW-15784] [C++][Python] Parallel parquet file reading disabled with single file reads

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 7.0.0
• Fix Version/s: 7.0.2, 8.0.0
• Component/s: C++, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/31227
• Assignee: Weston Pace
• Reporter: Weston Pace
• Created: 25/Feb/22 02:36
• Updated: 11/Jan/23 11:39
• Resolved: 01/Mar/22 14:12

Description

There is a flag enable_parallel_column_conversion which was passed down from python to C++ when reading parquet datasets which controlled whether we would read columns in parallel. This was allowed for single files but not for reading multiple files. This was an old check to help prevent nested deadlock.

Nested deadlock is no longer an issue and the flag was mostly inert once we removed the synchronous scanner.

Unfortunately, when we removed the synchronous scanner we forgot to remove this flag and the result was that a single-file read ended up disabling parallelism.

Issue Links

• links to: GitHub Pull Request #12514