Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
6.0.1
Description
S3Filesystem operations can fail with opaque curl errors that are hard to debug from R, as I could see no way of accessing what URL or header information was being sent even when we enter the R debugger. We are left with relatively opaque errors such as:
Error: IOError: When resolving region for bucket 'analysis': AWS Error [code 99]: curlCode: 6, Couldn't resolve host name
which could have a whole host of underlying causes, from incorrect URL construction (e.g. incorrectly specified endpoint_override, default_region, etc) to other network issues https certificate failure or network connectivity problems.
In any event, a common source of this seems to be collisions with AWS credentials in environmental variables or elsewhere interfering with the intended credentials (which may be fine with 'default' values – e.g. like when accessing a MINIO bucket we do not need a "region" (actually it's not needed in AWS as it can be attained from the endpoint; or at least that is the argument given by MINIO).
In a recent case, I'd already made sure I didn't have any AWS_DEFAULT_REGION or AWS_S3_ENDPOINT env var set, but still kept getting this error on a MINIO bucket on one machine , but not others.
Neal pointed out to me the existence of `AWS_EC2_METADATA_DISABLED`, and indeed, setting that to TRUE resolved the issue. Others may also be unaware of this option, it might be good to highlight in the S3Filesystem vignette and/or docs.
Attachments
Issue Links
- links to
