DeltaCloud
  1. DeltaCloud
  2. DTACLOUD-196

Deltacloud API displays buckets/blobs belonging to different locations(endpoints)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Component/s: Client (Ruby), Server
    • Labels:
      None
    • Environment:
      Deltacloud API 0.5.0
      OS - Fedora 15, 16, RHEL, 6.1
      Setup - EC2
      commit - c13da7b50bbbbdc529b42207f58dbe5099006ad1

      Description

      Deltacloud API displays buckets/blobs belonging to different locations(endpoints)

      Reproduction Steps:
      ==============
      1. Change the provider account from default to say ap-northeast-1
      2. Click on Buckets and create a bucket by providing the location as ap-northeast-1 (i.e, endpoint)
      3. Click on the created bucket and click on create blob
      4. Enter the below mentioned blob details and create the blob
      Blob Name: Ramesh-Test-Blob
      Blob Data: 2.txt (empty file uploaded from the home directory)
      Metadata Key: rananda
      Metadata Value: 2

      Scenario - 1: ( for any other providers other than us-east-1)
      ==========
      a. Now change the provider to say ap-southeast-1.
      b. Click on the buckets

      Expected Result:
      =============
      1. Should not display bucket which are created in the other providers or endpoints(but displays in this case)
      2. Click on the bucket

      Actual Results:
      ============
      1. Displays buckets belonging to different endpoints
      2. Clicking on the bucket name throws error stating "Deltacloud::ExceptionHandler::ProviderError - bad URI(is not URI?):"
      Refer "Error-southeast.png" and "ap-southeast-1-console-log.txt" for further information

      Scenario - 2: (for us-east-1 provider)
      ===========
      a. Now change the provider to say us-east-1
      b. Click on the buckets.

      Expected Result:
      =============
      1. Should not display bucket which are created in the other providers or endpoints(but displays in this case)
      2. Click on the bucket.
      3. Click on the blobs link which was created above

      Actual Results:
      ============
      1. Displays buckets belonging to different endpoints
      2. Clicking on the bucket displays the bucket details and also show the attached blob to this bucket
      3. Clicking on the blob link throws error stating "Deltacloud::ExceptionHandler::ProviderError - NilClass is not a valid input stream. It must walk like either a String, an IO, or a Source."
      Refer "Error-useast1.png" and "us-east-1-console-log.txt" for further details

      1. 0001-Fix-for-JIRA-DTACLOUD_196-Use-correct-endpoint-when.patch
        5 kB
        Marios Andreou
      2. ap-southeast-1-console-log.txt
        17 kB
        Ramesh A
      3. Error-southeast.png
        80 kB
        Ramesh A
      4. Error-useast1.png
        134 kB
        Ramesh A
      5. us-east-1-console-log.txt
        26 kB
        Ramesh A

        Activity

        Hide
        Ramesh A added a comment -

        Verified and working fine on the commit - 8c89662e2394e2188be04ae0fbaa17a49acc4163

        Note: This requires latest release of aws-2.5.6.gem. QE tested by building up the aws gem locally by cloning their git repo (Thanks to Marios, for helping me to build and test this issue)

        Show
        Ramesh A added a comment - Verified and working fine on the commit - 8c89662e2394e2188be04ae0fbaa17a49acc4163 Note: This requires latest release of aws-2.5.6.gem. QE tested by building up the aws gem locally by cloning their git repo (Thanks to Marios, for helping me to build and test this issue)
        Hide
        Marios Andreou added a comment -

        try this - there is still one outstanding issue (streaming blob content) but this will have to be dealt with later - it is outside of deltacloud control.

        Show
        Marios Andreou added a comment - try this - there is still one outstanding issue (streaming blob content) but this will have to be dealt with later - it is outside of deltacloud control.
        Hide
        Marios Andreou added a comment -

        needs testing please

        Show
        Marios Andreou added a comment - needs testing please
        Hide
        Marios Andreou added a comment -

        Attached patch should fix this issue - needs testing please.

        Basically, there is a call to check where a bucket 'exists' (i.e. in which endpoint). After the initial 'GET' using whichever endpoint the client has requested, check the bucket location and if necessary re-initialise the client to use the correct endpoint. See https://github.com/appoxy/aws/pull/113 for context.

        Show
        Marios Andreou added a comment - Attached patch should fix this issue - needs testing please. Basically, there is a call to check where a bucket 'exists' (i.e. in which endpoint). After the initial 'GET' using whichever endpoint the client has requested, check the bucket location and if necessary re-initialise the client to use the correct endpoint. See https://github.com/appoxy/aws/pull/113 for context.
        Hide
        Marios Andreou added a comment -

        For issue #2 above (getting a specific bucket when using a :provider (endpoint) other than the one in which the bucket exists and ) the fix has been pulled into the aws gem https://github.com/appoxy/aws/pull/110 - will have to wait for new version before updating our gemspec (currently aws is @ v 2.5.6).

        marios

        Show
        Marios Andreou added a comment - For issue #2 above (getting a specific bucket when using a :provider (endpoint) other than the one in which the bucket exists and ) the fix has been pulled into the aws gem https://github.com/appoxy/aws/pull/110 - will have to wait for new version before updating our gemspec (currently aws is @ v 2.5.6). marios
        Hide
        Marios Andreou added a comment -

        please review attached patch but leave issue open for now

        Show
        Marios Andreou added a comment - please review attached patch but leave issue open for now
        Hide
        Marios Andreou added a comment -

        OK. I think this ticket is about 3 issues:

        1. (ticket title): Deltacloud API displays buckets/blobs belonging to different locations(endpoints)
        2. getting a specific bucket when using a :provider (endpoint) other than the one in which the bucket exists and
        3. getting a specific blob when using a :provider (endpoint) other than the one in which the blob's bucket exists.

        =========================

        1. I don't think this is an issue - at least, I can't find anywhere in the AWS API that states the GET Service (i.e. list buckets) call will return only those buckets that were created in the request endpoint (http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTServiceGET.html)

        2. This is a problem with the aws gem. In short, AWS respond with a PermanentRedirect in such cases, providing the endpoint to which the bucket request should be made. I've sent a fix and pull request to aws for this: https://github.com/appoxy/aws/pull/110

        3. This is a (potentially at least) problem with AWS. It is related to the PermanentRedirect issue, however, getting blob details (without the blob data itself) is done with a HEAD request. Responses to HEAD do not contain any body, and so I don't know how the aws gem can handle this (i.e. there is no way of knowing where the new request should be made). I've posted a question on the aws forum here: https://forums.aws.amazon.com/thread.jspa?threadID=93141

        A potential fix is to set the endpoint to DEFAULT_REGION when a specific bucket/blob is requested. This however means that although user has requested an endpoint of e.g. 's3-ap-southeas-1' they'd actually get their call routed via s3.amazonaws.com. I don't know if this is acceptable. I attach the patch here for review/comments,

        marios

        Show
        Marios Andreou added a comment - OK. I think this ticket is about 3 issues: 1. (ticket title): Deltacloud API displays buckets/blobs belonging to different locations(endpoints) 2. getting a specific bucket when using a :provider (endpoint) other than the one in which the bucket exists and 3. getting a specific blob when using a :provider (endpoint) other than the one in which the blob's bucket exists. ========================= 1. I don't think this is an issue - at least, I can't find anywhere in the AWS API that states the GET Service (i.e. list buckets) call will return only those buckets that were created in the request endpoint ( http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTServiceGET.html ) 2. This is a problem with the aws gem. In short, AWS respond with a PermanentRedirect in such cases, providing the endpoint to which the bucket request should be made. I've sent a fix and pull request to aws for this: https://github.com/appoxy/aws/pull/110 3. This is a (potentially at least) problem with AWS. It is related to the PermanentRedirect issue, however, getting blob details (without the blob data itself) is done with a HEAD request. Responses to HEAD do not contain any body, and so I don't know how the aws gem can handle this (i.e. there is no way of knowing where the new request should be made). I've posted a question on the aws forum here: https://forums.aws.amazon.com/thread.jspa?threadID=93141 A potential fix is to set the endpoint to DEFAULT_REGION when a specific bucket/blob is requested. This however means that although user has requested an endpoint of e.g. 's3-ap-southeas-1' they'd actually get their call routed via s3.amazonaws.com. I don't know if this is acceptable. I attach the patch here for review/comments, marios

          People

          • Assignee:
            Ramesh A
            Reporter:
            Ramesh A
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development