Details

      Description

      When using a BlobStore configured for the 'aws-s3' provider and with an explicit region ('eu-west-1' in this case), BlobStore.createContainerInLocation(Location location, String container) fails with a 400 when attempting to create an already existing bucket on that location.

      The error message from AWS claims the cause to be AuthorizationHeaderMalformed, and says that 'eu-west-1' was expected, but that 'us-east-1' was supplied in the PUT request.

      I have confirmed that 'eu-west-1' is, in fact, given to the method call, but it does not seem to be respected when forming and signing the request.

      This does not happen on 1.9.x.

      From the debugging i have done a likely culprit seems to be Aws4SignerForAuthorizationHeader which resolves a ServiceAndRegion instance, which in turn seem to attempt parsing the region from the hostname URL (AWSHostNameUtils.parseRegionName(..)). I cannot see the region set for the provider being respected in this case.

        Activity

        Hide
        gaul Andrew Gaul added a comment - - edited

        Halvdan Hoem Grelland Can you provide a complete test case? I successfully executed:

        Location location = Iterables.tryFind(blobStore.listAssignableLocations(),
                LocationPredicates.idEquals(Region.EU_CENTRAL_1)).orNull();
        assertNotNull(location);
        blobStore.createContainerInLocation(location, containerName);
        blobStore.createContainerInLocation(location, containerName);
        String blobName = "test-blob";
        Blob blob = blobStore.blobBuilder(blobName).payload("something").build();
        blobStore.putBlob(containerName, blob);
        blobStore.removeBlob(containerName, blobName);
        

        I verified the correct region was set by disabling virtual host buckets and examining the wire logs:

        >> PUT https://s3.amazonaws.com/gaul-blobstore2eu HTTP/1.1
        >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=07f890b1f2629b2cc49153aeab3a190fc5874d529d4bb890077dc4702bfc458f
        
        << HTTP/1.1 200 OK
        
        >> PUT https://s3.amazonaws.com/gaul-blobstore2eu HTTP/1.1
        >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=07f890b1f2629b2cc49153aeab3a190fc5874d529d4bb890077dc4702bfc458f
        
        << HTTP/1.1 409 Conflict
        << x-amz-bucket-region: eu-central-1
        
        >> GET https://s3.amazonaws.com/gaul-blobstore2eu?location HTTP/1.1
        >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2321d91d99430526251e82bbaab748a97cb0cc1a1d6bfce763a5b2b20a10b148
        
        << HTTP/1.1 200 OK
        
        >> PUT https://s3-eu-central-1.amazonaws.com/gaul-blobstore2eu/test-blob HTTP/1.1
        >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/eu-central-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=b96ac1928ba38a063caa7c2a80f484007dce843eae6a8bc35ade1abd134b967e
        
        << HTTP/1.1 200 OK
        
        >> DELETE https://s3-eu-central-1.amazonaws.com/gaul-blobstore2eu/test-blob HTTP/1.1
        >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=25f6f0b552266b790a996577321620349ce654a3cd66932ea899f863966f06cb
        
        << HTTP/1.1 204 No Content
        
        Show
        gaul Andrew Gaul added a comment - - edited Halvdan Hoem Grelland Can you provide a complete test case? I successfully executed: Location location = Iterables.tryFind(blobStore.listAssignableLocations(), LocationPredicates.idEquals(Region.EU_CENTRAL_1)).orNull(); assertNotNull(location); blobStore.createContainerInLocation(location, containerName); blobStore.createContainerInLocation(location, containerName); String blobName = "test-blob" ; Blob blob = blobStore.blobBuilder(blobName).payload( "something" ).build(); blobStore.putBlob(containerName, blob); blobStore.removeBlob(containerName, blobName); I verified the correct region was set by disabling virtual host buckets and examining the wire logs: >> PUT https: //s3.amazonaws.com/gaul-blobstore2eu HTTP/1.1 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=07f890b1f2629b2cc49153aeab3a190fc5874d529d4bb890077dc4702bfc458f << HTTP/1.1 200 OK >> PUT https: //s3.amazonaws.com/gaul-blobstore2eu HTTP/1.1 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=07f890b1f2629b2cc49153aeab3a190fc5874d529d4bb890077dc4702bfc458f << HTTP/1.1 409 Conflict << x-amz-bucket-region: eu-central-1 >> GET https: //s3.amazonaws.com/gaul-blobstore2eu?location HTTP/1.1 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2321d91d99430526251e82bbaab748a97cb0cc1a1d6bfce763a5b2b20a10b148 << HTTP/1.1 200 OK >> PUT https: //s3-eu-central-1.amazonaws.com/gaul-blobstore2eu/test-blob HTTP/1.1 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/eu-central-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=b96ac1928ba38a063caa7c2a80f484007dce843eae6a8bc35ade1abd134b967e << HTTP/1.1 200 OK >> DELETE https: //s3-eu-central-1.amazonaws.com/gaul-blobstore2eu/test-blob HTTP/1.1 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=25f6f0b552266b790a996577321620349ce654a3cd66932ea899f863966f06cb << HTTP/1.1 204 No Content
        Hide
        timuralp Timur Alperovich added a comment -

        I'm running into a similar problem when attempting to list contents of a non-US-Standard bucket.

        Here is the simple failing example (fails with jclouds 2.0.0):

        public class TestMain {
            public static void main(String[] args) throws Exception {
                BlobStoreContext context = ContextBuilder.newBuilder("aws-s3")
                        .credentials("AWS-ACCESS-KEY-ID", "SECRET-ACCESS-KEY")
                        .buildView(BlobStoreContext.class);
                BlobStore blobStore = context.getBlobStore();
                for (StorageMetadata meta: blobStore.list("EU-West-1-Bucket")) {
                    System.out.println(meta.getName());
                }
            }
        }
        

        The resulting error is:
        message='The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1''

        I suspect jclouds needs to check the bucket location before submitting the request. I haven't figured out how to express that.

        Show
        timuralp Timur Alperovich added a comment - I'm running into a similar problem when attempting to list contents of a non-US-Standard bucket. Here is the simple failing example (fails with jclouds 2.0.0): public class TestMain { public static void main( String [] args) throws Exception { BlobStoreContext context = ContextBuilder.newBuilder( "aws-s3" ) .credentials( "AWS-ACCESS-KEY-ID" , "SECRET-ACCESS-KEY" ) .buildView(BlobStoreContext.class); BlobStore blobStore = context.getBlobStore(); for (StorageMetadata meta: blobStore.list( "EU-West-1-Bucket" )) { System .out.println(meta.getName()); } } } The resulting error is: message='The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'' I suspect jclouds needs to check the bucket location before submitting the request. I haven't figured out how to express that.
        Hide
        gaul Andrew Gaul added a comment -

        This same code works for me using 2.1.0-SNAPSHOT. Abbreviated wire logs:

        >> GET https://gaultest-eu2.s3.amazonaws.com/?location HTTP/1.1
        >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=1a5d2a283b463caa68eb10960f012583a7564eb5d34bb0c5237a3fe532c15314
        << HTTP/1.1 200 OK
        
        >> GET https://gaultest-eu2.s3-eu-west-1.amazonaws.com/?delimiter=/ HTTP/1.1
        >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8b9a629cb5fb973bec6b625bf5ad9a0d24dda45716ede3116be4d3318558760f
        << HTTP/1.1 200 OK
        << x-amz-bucket-region: eu-west-1
        
        Show
        gaul Andrew Gaul added a comment - This same code works for me using 2.1.0-SNAPSHOT. Abbreviated wire logs: >> GET https: //gaultest-eu2.s3.amazonaws.com/?location HTTP/1.1 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=1a5d2a283b463caa68eb10960f012583a7564eb5d34bb0c5237a3fe532c15314 << HTTP/1.1 200 OK >> GET https: //gaultest-eu2.s3-eu-west-1.amazonaws.com/?delimiter=/ HTTP/1.1 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAITMZ4YFWHBLA4KMA/20170111/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8b9a629cb5fb973bec6b625bf5ad9a0d24dda45716ede3116be4d3318558760f << HTTP/1.1 200 OK << x-amz-bucket-region: eu-west-1
        Hide
        timuralp Timur Alperovich added a comment -

        Thanks for checking. I can reproduce this consistently with 2.1.0-SNAPSHOT, as well. I'll attempt to debug this locally. The specific difference that I'm observing is the response to the bucket location request against the EU bucket:

        >> GET https://timur-test-ireland.s3.amazonaws.com/?location HTTP/1.1
        >> Host: timur-test-ireland.s3.amazonaws.com
        >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        >> X-Amz-Date: 20170111T233842Z
        >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIT4YWLZX26RHCRKQ/20170111/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2ca401371c162bd8e08fd161c2c330d5f887d08ae4f5996d3d2040e9032be86d
        << HTTP/1.1 400 Bad Request
        << Transfer-Encoding: chunked
        << Server: AmazonS3
        << Connection: close
        << x-amz-request-id: 580278C04C186502
        << x-amz-id-2: 1lEvRCBWBmb3Tt/xe8T7u7HVuUNCT3hNWfzfd3VwKFWhmE56rf2y2iaRk1UCT1YkMK78Q+hpKlI=
        << Date: Wed, 11 Jan 2017 23:37:57 GMT
        << Content-Type: application/xml
        org.jclouds.aws.AWSResponseException: request GET https://timur-test-ireland.s3.amazonaws.com/?location HTTP/1.1 failed with code 400, error: AWSError{requestId='580278C04C186502', requestToken='1lEvRCBWBmb3Tt/xe8T7u7HVuUNCT3hNWfzfd3VwKFWhmE56rf2y2iaRk1UCT1YkMK78Q+hpKlI=', code='AuthorizationHeaderMalformed', message='The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'', context='{Region=eu-west-1, HostId=1lEvRCBWBmb3Tt/xe8T7u7HVuUNCT3hNWfzfd3VwKFWhmE56rf2y2iaRk1UCT1YkMK78Q+hpKlI=}'}
        
        Show
        timuralp Timur Alperovich added a comment - Thanks for checking. I can reproduce this consistently with 2.1.0-SNAPSHOT, as well. I'll attempt to debug this locally. The specific difference that I'm observing is the response to the bucket location request against the EU bucket: >> GET https: //timur-test-ireland.s3.amazonaws.com/?location HTTP/1.1 >> Host: timur-test-ireland.s3.amazonaws.com >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 >> X-Amz-Date: 20170111T233842Z >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIT4YWLZX26RHCRKQ/20170111/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2ca401371c162bd8e08fd161c2c330d5f887d08ae4f5996d3d2040e9032be86d << HTTP/1.1 400 Bad Request << Transfer-Encoding: chunked << Server: AmazonS3 << Connection: close << x-amz-request-id: 580278C04C186502 << x-amz-id-2: 1lEvRCBWBmb3Tt/xe8T7u7HVuUNCT3hNWfzfd3VwKFWhmE56rf2y2iaRk1UCT1YkMK78Q+hpKlI= << Date: Wed, 11 Jan 2017 23:37:57 GMT << Content-Type: application/xml org.jclouds.aws.AWSResponseException: request GET https: //timur-test-ireland.s3.amazonaws.com/?location HTTP/1.1 failed with code 400, error: AWSError{requestId='580278C04C186502', requestToken='1lEvRCBWBmb3Tt/xe8T7u7HVuUNCT3hNWfzfd3VwKFWhmE56rf2y2iaRk1UCT1YkMK78Q+hpKlI=', code='AuthorizationHeaderMalformed', message='The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'', context='{Region=eu-west-1, HostId=1lEvRCBWBmb3Tt/xe8T7u7HVuUNCT3hNWfzfd3VwKFWhmE56rf2y2iaRk1UCT1YkMK78Q+hpKlI=}'}
        Hide
        timuralp Timur Alperovich added a comment -

        Reading more about this, I suspect I'm bumping into an issue similar to the one reported here: https://github.com/aws/aws-sdk-js/issues/462.

        Specifically, when I attempt to resolve the bucket in question, I get back the following for a bucket in US-West-2:

        timur-test.s3.amazonaws.com is an alias for s3-us-west-2-w.amazonaws.com.
        s3-us-west-2-w.amazonaws.com has address 54.231.184.246
        

        At this point the GetBucketLocation request, which is meant to go to timur-test.s3.amazonaws.com in us-east-1 gets re-routed to a us-west-2 endpoint and results in the error I'm seeing. I investigated how the boto3 library handles this and they use bucket in-path requests. I'm going to submit a pull request for jclouds to do the same. For the particular failure, I suspect the difference in networking resulted in my issue, which was not reproducible by Andrew.

        Show
        timuralp Timur Alperovich added a comment - Reading more about this, I suspect I'm bumping into an issue similar to the one reported here: https://github.com/aws/aws-sdk-js/issues/462 . Specifically, when I attempt to resolve the bucket in question, I get back the following for a bucket in US-West-2: timur-test.s3.amazonaws.com is an alias for s3-us-west-2-w.amazonaws.com. s3-us-west-2-w.amazonaws.com has address 54.231.184.246 At this point the GetBucketLocation request, which is meant to go to timur-test.s3.amazonaws.com in us-east-1 gets re-routed to a us-west-2 endpoint and results in the error I'm seeing. I investigated how the boto3 library handles this and they use bucket in-path requests. I'm going to submit a pull request for jclouds to do the same. For the particular failure, I suspect the difference in networking resulted in my issue, which was not reproducible by Andrew.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 98ea9174770693035158b5efba8efef1abe4a934 in jclouds's branch refs/heads/master from Timur Alperovich
        [ https://git-wip-us.apache.org/repos/asf?p=jclouds.git;h=98ea917 ]

        GetBucketLocation should use path-style requests.

        When making a GetBucketLocation request, Amazon may route the request
        to the bucket region. When making it with v4 signer, the request may
        fail because of the region mismatch. Concretely, a request to
        test.s3.amazonaws.com may resolve to s3-us-west-2-w.amazonaws.com. The
        request itself is prepared for the us-east-1 region (s3.amazonaws.com
        endpoint), but then fails when the DNS resolution points to a
        us-west-2 endpoint.

        Bucket-in-path works around this for the GetBucketLocation requests.
        That means that every GetBucketLocation request will be of the form:
        https://s3.amazonaws.com/

        {bucket}

        ?location. This ensures that jclouds
        requests will not be subjected to Amazon's routing/DNS pointers.

        Fixes: JCLOUDS-1213

        Show
        jira-bot ASF subversion and git services added a comment - Commit 98ea9174770693035158b5efba8efef1abe4a934 in jclouds's branch refs/heads/master from Timur Alperovich [ https://git-wip-us.apache.org/repos/asf?p=jclouds.git;h=98ea917 ] GetBucketLocation should use path-style requests. When making a GetBucketLocation request, Amazon may route the request to the bucket region. When making it with v4 signer, the request may fail because of the region mismatch. Concretely, a request to test.s3.amazonaws.com may resolve to s3-us-west-2-w.amazonaws.com. The request itself is prepared for the us-east-1 region (s3.amazonaws.com endpoint), but then fails when the DNS resolution points to a us-west-2 endpoint. Bucket-in-path works around this for the GetBucketLocation requests. That means that every GetBucketLocation request will be of the form: https://s3.amazonaws.com/ {bucket} ?location. This ensures that jclouds requests will not be subjected to Amazon's routing/DNS pointers. Fixes: JCLOUDS-1213
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 3668a9905b4667fa6b9ca47e894a96abd182ea93 in jclouds's branch refs/heads/2.0.x from Timur Alperovich
        [ https://git-wip-us.apache.org/repos/asf?p=jclouds.git;h=3668a99 ]

        GetBucketLocation should use path-style requests.

        When making a GetBucketLocation request, Amazon may route the request
        to the bucket region. When making it with v4 signer, the request may
        fail because of the region mismatch. Concretely, a request to
        test.s3.amazonaws.com may resolve to s3-us-west-2-w.amazonaws.com. The
        request itself is prepared for the us-east-1 region (s3.amazonaws.com
        endpoint), but then fails when the DNS resolution points to a
        us-west-2 endpoint.

        Bucket-in-path works around this for the GetBucketLocation requests.
        That means that every GetBucketLocation request will be of the form:
        https://s3.amazonaws.com/

        {bucket}

        ?location. This ensures that jclouds
        requests will not be subjected to Amazon's routing/DNS pointers.

        Fixes: JCLOUDS-1213

        Show
        jira-bot ASF subversion and git services added a comment - Commit 3668a9905b4667fa6b9ca47e894a96abd182ea93 in jclouds's branch refs/heads/2.0.x from Timur Alperovich [ https://git-wip-us.apache.org/repos/asf?p=jclouds.git;h=3668a99 ] GetBucketLocation should use path-style requests. When making a GetBucketLocation request, Amazon may route the request to the bucket region. When making it with v4 signer, the request may fail because of the region mismatch. Concretely, a request to test.s3.amazonaws.com may resolve to s3-us-west-2-w.amazonaws.com. The request itself is prepared for the us-east-1 region (s3.amazonaws.com endpoint), but then fails when the DNS resolution points to a us-west-2 endpoint. Bucket-in-path works around this for the GetBucketLocation requests. That means that every GetBucketLocation request will be of the form: https://s3.amazonaws.com/ {bucket} ?location. This ensures that jclouds requests will not be subjected to Amazon's routing/DNS pointers. Fixes: JCLOUDS-1213

          People

          • Assignee:
            timuralp Timur Alperovich
            Reporter:
            halvdanhg Halvdan Hoem Grelland
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development