[ARROW-5825] [Python] Exceptions swallowed in ParquetManifest._visit_directories

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Python
• Labels: dataset-parquet-read parquet
• External issue URL: https://github.com/apache/arrow/issues/22244
• Assignee: Unassigned
• Reporter: George Sakkis
• Created: 02/Jul/19 19:51
• Updated: 11/Jan/23 07:42

Description

ParquetManifest._visit_directories uses a ThreadPoolExecutor to visit partitioned parquet datasets concurrently, it waits for them to finish but doesn't check if the respective futures have failed or not. This is quite tricky to detect and debug as an exception is either raised later as a a side-effect or (perhaps worse) it passes silently.

Observed on 0.12.1 but appears to be on latest master too.