[ARROW-16619] [R] Support compression + R connection (URL with .gz file)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.0.0
• Component/s: R
• External issue URL: https://github.com/apache/arrow/issues/31972
• Assignee: Neal Richardson
• Reporter: Carl Boettiger
• Created: 19/May/22 19:25
• Updated: 11/Jan/23 11:45
• Resolved: 03/Aug/22 12:13

Description

Currently, remote access to data (particularly lazy read, an immensely powerful arrow ability) only works for data in an S3-compliant object store (though I know Azure support is in the works).  It would be really fantastic if we could have remote access over HTTPS (I think this already works on the python side thanks to fsspec). 

For example, this fails in arrow but works in readr:

arrow::read_csv_arrow("https://data.ecoforecast.org/targets/aquatics/aquatics-targets.csv.gz")
 
readr::read_csv("https://data.ecoforecast.org/targets/aquatics/aquatics-targets.csv.gz")

I think this ability would be even more compelling in `open_dataset()`, since it opens up for us all the power of lazy read access.  Most servers support curl range requests so it seems this should be possible.  (We can already do something similar from duckdb+R, but only after manually opting in the http extension and only for parquet).

Issue Links

• duplicates: ARROW-7594 [C++] Implement HTTP and FTP file systems
• duplicates: ARROW-14998 [R] Support for HTTPS Filesystem access
• is fixed by: ARROW-16612 [R] Fix compression inference from filename