[ARROW-4406] Ignore "*_$folder$" files on S3

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: Python
• Labels: easyfix pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/20969
• Assignee: George Sakkis
• Reporter: George Sakkis
• Created: 28/Jan/19 17:52
• Updated: 11/Jan/23 07:33
• Resolved: 07/Feb/19 12:44

Description

Currently reading parquet files generated by Hadoop (EMR) from S3 fails with "ValueError: Found files in an intermediate directory" because of the _$folder$ empty files. 

The fix should be easy, just an extra condition in ParquetManifest._should_silently_exclude.

Issue Links

• links to: GitHub Pull Request #3577