[ARROW-9854] [R] Support reading/writing data to/from S3

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: R
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25892
• Assignee: Neal Richardson
• Reporter: Neal Richardson
• Created: 25/Aug/20 18:05
• Updated: 11/Jan/23 08:09
• Resolved: 10/Sep/20 21:59

Description

Current S3 support is limited to (1) being able to instantiate an S3FileSystem object, primarily from a URI, and (2) ability to open_dataset from an S3 URI. Before widely declaring that we support S3 in R, we should be able to:

• download dataset (i.e. copy files/directory recursively)
• read_parquet/feather/etc. from S3 (use FileSystem->OpenInputFile(path))
• write_$FORMAT via FileSystem->OpenOutputStream(path)
• write_dataset
• for linux, an argument to install_arrow to help, assuming you've installed aws-sdk-cpp already (turn on ARROW_S3, AWSSDK_SOURCE=SYSTEM)
• testing with minio on CI
• set up a real test bucket and user for e2e testing
• update docs and vignettes

Issue Links

• links to: GitHub Pull Request #8058