[ARROW-6713] [Python] Getting "ArrowIOError: Corrupted file, smaller than file footer" when reading large number of parquet files to ParquetDataset()

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Labels: parquet
• External issue URL: https://github.com/apache/arrow/issues/23057
• Assignee: Unassigned
• Reporter: Harini Kannan
• Created: 26/Sep/19 18:37
• Updated: 11/Jan/23 07:48
• Resolved: 16/Oct/19 20:40

Description

When trying to read a large number of parquet files (> 600) into ParquetDataset(), getting the error: 

ArrowIOError: Corrupted file, smaller than file footer.

 

This could be related to this issue: https://issues.apache.org/jira/browse/ARROW-3424

Note:

-This works fine for small number of (< 245 to be exact, not sure if this helps) parquet files.