Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-15127

[R] More visible documentation of AWS_EC2_METADATA_DISABLED=TRUE

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 6.0.1
    • 7.0.0
    • R

    Description

      S3Filesystem operations can fail with opaque curl errors that are hard to debug from R, as I could see no way of accessing what URL or header information was being sent even when we enter the R debugger.  We are left with relatively opaque errors such as:

      Error: IOError: When resolving region for bucket 'analysis': AWS Error [code 99]: curlCode: 6, Couldn't resolve host name 

      which could have a whole host of underlying causes, from incorrect URL construction (e.g. incorrectly specified endpoint_override, default_region, etc) to other network issues https certificate failure or network connectivity problems.

      In any event, a common source of this seems to be collisions with AWS credentials in environmental variables or elsewhere interfering with the intended credentials (which may be fine with 'default' values – e.g. like when accessing a MINIO bucket we do not need a "region" (actually it's not needed in AWS as it can be attained from the endpoint; or at least that is the argument given by MINIO).

      In a recent case, I'd already made sure I didn't have any AWS_DEFAULT_REGION or AWS_S3_ENDPOINT env var set, but still kept getting this error on a MINIO bucket on one machine , but not others. 

      Neal pointed out to me the existence of `AWS_EC2_METADATA_DISABLED`, and indeed, setting that to TRUE resolved the issue.  Others may also be unaware of this option, it might be good to highlight in the S3Filesystem vignette and/or docs.

       

       

      Attachments

        Issue Links

          Activity

            People

              thisisnic Nicola Crane
              cboettig Carl Boettiger
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 20m
                  2h 20m

                  Slack

                    Issue deployment