Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: core
    • Labels:
      None

      Description

      Each time a BlobStore context is created, so are 2 more thread pools. Moreover, there are remote calls to obtain provider metadata which are redundantly parsed. Like ComputeCache, we should have a cache used in BlobStoreContextBuilder which is closed when the whirr processes do.

      1. BlobStoreContextBuilder.java
        8 kB
        Adrian Cole
      2. BlobStoreContextBuilder.java
        7 kB
        Adrian Cole
      3. WHIRR-362.patch
        8 kB
        Andrei Savu

        Issue Links

          Activity

          Hide
          Adrian Cole (Inactive) added a comment -

          save as the following once WHIRR-361 is applied
          ./core/src/main/java/org/apache/whirr/service/BlobStoreContextBuilder.java

          Show
          Adrian Cole (Inactive) added a comment - save as the following once WHIRR-361 is applied ./core/src/main/java/org/apache/whirr/service/BlobStoreContextBuilder.java
          Hide
          Adrian Cole (Inactive) added a comment -

          tested, and noted log output where only a single BlobStoreContext is created during BlobCacheTest

          Show
          Adrian Cole (Inactive) added a comment - tested, and noted log output where only a single BlobStoreContext is created during BlobCacheTest
          Hide
          Adrian Cole (Inactive) added a comment -

          Note this also allows you to pass context properties to BlobStore, something not supported prior. For example, the following configuration properly configures s3 to the eu region:

          whirr.blobstore-location-id=EU
          jclouds.aws-s3.endpoint=https://s3-eu-west-1.amazonaws.com

          Show
          Adrian Cole (Inactive) added a comment - Note this also allows you to pass context properties to BlobStore, something not supported prior. For example, the following configuration properly configures s3 to the eu region: whirr.blobstore-location-id=EU jclouds.aws-s3.endpoint= https://s3-eu-west-1.amazonaws.com
          Hide
          Adrian Cole (Inactive) added a comment -

          new version that properly handles user-configured jclouds providers like walrus

          Show
          Adrian Cole (Inactive) added a comment - new version that properly handles user-configured jclouds providers like walrus
          Hide
          Adrian Cole (Inactive) added a comment -

          new patch tested against the ECC walrus endpoint:

          whirr.blobstore-provider=walrus
          whirr.blobstore-identity=XXXX
          whirr.blobstore-credential=XXXX
          jclouds.walrus.endpoint=http://partnercloud.eucalyptus.com:8773/services/Walrus
          jclouds.walrus.iso3166-codes=US-CA

          2011-08-13 14:14:23,568 INFO [org.apache.whirr.service.BlobStoreContextBuilder$Cache] (main) created new BlobStoreContext [id=walrus, endpoint=http://partnercloud.eucalyptus.com:8773/services/Walrus, apiVersion=Walrus-1.6, identity=5vOVwiDWcArl0zzqDLlaQPXdIhynAxz9ZSLapw, iso3166Codes=[US-CA]]
          2011-08-13 14:14:26,111 INFO [org.apache.whirr.util.BlobCache] (main) Created blob cache container '3kq6ii50pfaz' located in 'null'
          2011-08-13 14:14:27,463 INFO [org.apache.whirr.util.BlobCache] (main) Uploading 'whirr7960498924063392388.txt' to '3kq6ii50pfaz' blob cache.
          2011-08-13 14:14:32,950 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) (mkdir -p /tmp && cd /tmp && [ ! -f whirr7960498924063392388.txt ] && curl -C - -s -q -L --connect-timeout 10 --max-time 600 -X GET -s --retry 20 -H "Date: Sat, 13 Aug 2011 13:14:23 GMT" -H "Authorization: AWS 5vOVwiDWcArl0zzqDLlaQPXdIhynAxz9ZSLapw:frDEUvWM6FUAzGOelP8iEOBqLwQ=" http://partnercloud.eucalyptus.com:8773/services/Walrus/3kq6ii50pfaz/whirr7960498924063392388.txt >whirr7960498924063392388.txt)

          2011-08-13 14:14:32,950 INFO [org.apache.whirr.util.BlobCache] (main) Removing blob cache '3kq6ii50pfaz'
          2011-08-13 14:14:34,659 INFO [org.apache.whirr.util.BlobCache] (main) Created blob cache container 'jhp3iaisleos' located in 'null'
          2011-08-13 14:14:34,906 INFO [org.apache.whirr.util.BlobCache] (main) Uploading 'whirr6631307362110065793.txt' to 'jhp3iaisleos' blob cache.
          2011-08-13 14:15:15,087 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) (mkdir -p /tmp && cd /tmp && [ ! -f whirr6631307362110065793.txt ] && curl -C - -s -q -L --connect-timeout 10 --max-time 600 -X GET -s --retry 20 -H "Date: Sat, 13 Aug 2011 13:14:23 GMT" -H "Authorization: AWS 5vOVwiDWcArl0zzqDLlaQPXdIhynAxz9ZSLapw:vr5UFrrd7LcsqNRxWfB2RqoFOho=" http://partnercloud.eucalyptus.com:8773/services/Walrus/jhp3iaisleos/whirr6631307362110065793.txt >whirr6631307362110065793.txt)

          2011-08-13 14:15:15,088 INFO [org.apache.whirr.util.BlobCache] (main) Removing blob cache 'jhp3iaisleos'
          2011-08-13 14:15:25,815 INFO [org.apache.whirr.service.ComputeCache] (main) created new ComputeServiceContext [id=aws-ec2, endpoint=https://ec2.us-east-1.amazonaws.com, apiVersion=2010-06-15, identity=067PW7Z9P0FNH7JDPE82, iso3166Codes=[US-VA, US-CA, IE, SG, JP-13]]
          2011-08-13 14:15:26,505 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) Created temporary container 'h1lzpns2x5rr'
          2011-08-13 14:15:26,769 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) Removing temporary container 'h1lzpns2x5rr'

          Show
          Adrian Cole (Inactive) added a comment - new patch tested against the ECC walrus endpoint: whirr.blobstore-provider=walrus whirr.blobstore-identity=XXXX whirr.blobstore-credential=XXXX jclouds.walrus.endpoint= http://partnercloud.eucalyptus.com:8773/services/Walrus jclouds.walrus.iso3166-codes=US-CA 2011-08-13 14:14:23,568 INFO [org.apache.whirr.service.BlobStoreContextBuilder$Cache] (main) created new BlobStoreContext [id=walrus, endpoint= http://partnercloud.eucalyptus.com:8773/services/Walrus , apiVersion=Walrus-1.6, identity=5vOVwiDWcArl0zzqDLlaQPXdIhynAxz9ZSLapw, iso3166Codes= [US-CA] ] 2011-08-13 14:14:26,111 INFO [org.apache.whirr.util.BlobCache] (main) Created blob cache container '3kq6ii50pfaz' located in 'null' 2011-08-13 14:14:27,463 INFO [org.apache.whirr.util.BlobCache] (main) Uploading 'whirr7960498924063392388.txt' to '3kq6ii50pfaz' blob cache. 2011-08-13 14:14:32,950 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) (mkdir -p /tmp && cd /tmp && [ ! -f whirr7960498924063392388.txt ] && curl -C - -s -q -L --connect-timeout 10 --max-time 600 -X GET -s --retry 20 -H "Date: Sat, 13 Aug 2011 13:14:23 GMT" -H "Authorization: AWS 5vOVwiDWcArl0zzqDLlaQPXdIhynAxz9ZSLapw:frDEUvWM6FUAzGOelP8iEOBqLwQ=" http://partnercloud.eucalyptus.com:8773/services/Walrus/3kq6ii50pfaz/whirr7960498924063392388.txt >whirr7960498924063392388.txt) 2011-08-13 14:14:32,950 INFO [org.apache.whirr.util.BlobCache] (main) Removing blob cache '3kq6ii50pfaz' 2011-08-13 14:14:34,659 INFO [org.apache.whirr.util.BlobCache] (main) Created blob cache container 'jhp3iaisleos' located in 'null' 2011-08-13 14:14:34,906 INFO [org.apache.whirr.util.BlobCache] (main) Uploading 'whirr6631307362110065793.txt' to 'jhp3iaisleos' blob cache. 2011-08-13 14:15:15,087 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) (mkdir -p /tmp && cd /tmp && [ ! -f whirr6631307362110065793.txt ] && curl -C - -s -q -L --connect-timeout 10 --max-time 600 -X GET -s --retry 20 -H "Date: Sat, 13 Aug 2011 13:14:23 GMT" -H "Authorization: AWS 5vOVwiDWcArl0zzqDLlaQPXdIhynAxz9ZSLapw:vr5UFrrd7LcsqNRxWfB2RqoFOho=" http://partnercloud.eucalyptus.com:8773/services/Walrus/jhp3iaisleos/whirr6631307362110065793.txt >whirr6631307362110065793.txt) 2011-08-13 14:15:15,088 INFO [org.apache.whirr.util.BlobCache] (main) Removing blob cache 'jhp3iaisleos' 2011-08-13 14:15:25,815 INFO [org.apache.whirr.service.ComputeCache] (main) created new ComputeServiceContext [id=aws-ec2, endpoint= https://ec2.us-east-1.amazonaws.com , apiVersion=2010-06-15, identity=067PW7Z9P0FNH7JDPE82, iso3166Codes= [US-VA, US-CA, IE, SG, JP-13] ] 2011-08-13 14:15:26,505 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) Created temporary container 'h1lzpns2x5rr' 2011-08-13 14:15:26,769 INFO [org.apache.whirr.util.integration.BlobCacheTest] (main) Removing temporary container 'h1lzpns2x5rr'
          Hide
          Andrei Savu added a comment -

          Created patch for this change. Tested on S3 & cloudfiles. The speed improvement is clearly visible.

          Show
          Andrei Savu added a comment - Created patch for this change. Tested on S3 & cloudfiles. The speed improvement is clearly visible.
          Hide
          Andrei Savu added a comment -

          I've just committed this.

          Show
          Andrei Savu added a comment - I've just committed this.

            People

            • Assignee:
              Adrian Cole (Inactive)
              Reporter:
              Adrian Cole (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development