Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14620

S3A authentication failure for regions other than us-east-1

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Works for Me
    • Affects Version/s: 2.7.3
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      hadoop fs s3a:// operations fail authentication for s3 buckets hosted in regions other than default us-east-1

      Steps to reproduce:

      1. create s3 bucket in eu-west-1
      2. Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run following command:
      hadoop --loglevel DEBUG  -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com  -ls  s3a://your-eu-west-1-hosted-bucket/ 
      

      Expected behaviour:
      You will see listing of the bucket

      Actual behaviour:
      You will get 403 Authentication Denied response for AWS S3.

      Reason is mismatch in string to sign as defined in http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html provided by hadoop and expected by AWS.

      If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes returned by AWS, you will see that AWS expects CanonicalizedResource to be in form /your-eu-west-1-hosted-bucket.s3.eu-west-1.amazonaws.com/.
      Hadoop provides it as /your-eu-west-1-hosted-bucket/

      Note that AWS documentation doesn't explicitly state that endpoint or full dns address should be appended to CanonicalizedResource however practice shows it is actually required.

      I've also submitted this to AWS for them to correct behaviour or documentation.

        Attachments

        1. s3-403.txt
          12 kB
          Ilya Fourmanov

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ilya.fourmanov Ilya Fourmanov
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: