[ARROW-15280] [R] Expose FileSystemFactoryOptions

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 6.0.1
• Fix Version/s: 9.0.0
• Component/s: R
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/30771
• Assignee: Neal Richardson
• Reporter: Bob Rudis
• Created: 07/Jan/22 10:59
• Updated: 11/Jan/23 11:36
• Resolved: 19/May/22 14:52

Description

ARROW-4406 notes that:

>Currently reading parquet files generated by Hadoop (EMR) from S3 fails with "ValueError: >Found files in an intermediate directory" because of the _$folder$ empty files.

This was fixed in the pyarrow but R still has this issue.

The R side does not seem to have something similar to:

  def _should_silently_exclude(self, file_name):
    return (file_name.endswith('.crc') or # Checksums
            file_name.endswith('_$folder$') or # HDFS directories in S3
            file_name.startswith('.') or # Hidden files starting with .
            file_name.startswith('_') or # Hidden files starting with _
            file_name in EXCLUDED_PARQUET_PATHS)

Issue Links

• fixes: ARROW-16720 [R] Cannot read datasets partitioned by columns starting with dots
• links to: GitHub Pull Request #13171