[ARROW-17201] [R] Cannot override environmental variables setting AWS S3 region

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 8.0.1
• Component/s: R
• External issue URL: https://github.com/apache/arrow/issues/32493
• Assignee: Unassigned
• Reporter: Carl Boettiger
• Created: 25/Jul/22 20:50
• Updated: 11/Jan/23 11:49

Description

If a user has set the AWS_DEFAULT_REGION, there is no way we can override that setting (especially to a null value) in a one-off call.  Consider the following example.  This fails:

library(arrow)
Sys.setenv(AWS_DEFAULT_REGION="data")

noaa <- s3_bucket("neon4cast-drivers/",
                  endpoint_override = "data.ecoforecast.org",
                  anonymous = TRUE) 

If the env var is not set or unset, this succeeds.  However, attempting to override the region does not help:

noaa <- s3_bucket("neon4cast-drivers/",
                  endpoint_override = "data.ecoforecast.org",
                  region = "us-east-1",
                  anonymous = TRUE) 

(nor does it help to set the region to "", or NULL. – note this is a MINIO host and so region is not needed anyway). 

Relatedly, one might expect that AWS_S3_ENDPOINT could be used instead of setting `endpoint_override`, but this does not seem to work either (lemme know if you want that in a separate issue thread).  At least on the plus side, the above code does not fail if AWS_S3_ENDPOINT is set to some other value than given in endpoint_override, so that is nice).