Whirr
  1. Whirr
  2. WHIRR-416

Enable lazy image fetching when the image-id is known

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: core
    • Labels:
      None

      Description

      "By default, the ComputeService will prefetch images from Alestic, Canonical, and RightScale. This allows you to query the images, which is great, if you don't know what you are looking for. However, if you already know the imageId you want, this is wasteful. We now provide the option to lazy-fetch images, which speed-ups execution."

      http://www.jclouds.org/documentation/userguide/using-ec2

      1. WHIRR-416.patch
        7 kB
        Andrei Savu
      2. WHIRR-416.patch
        8 kB
        Andrei Savu

        Activity

        Hide
        Andrei Savu added a comment -

        The jclouds documentation is a bit out of date, it should be AWSEC2Constants.PROPERTY_EC2_AMI_QUERY not EC2Constants

        Show
        Andrei Savu added a comment - The jclouds documentation is a bit out of date, it should be AWSEC2Constants.PROPERTY_EC2_AMI_QUERY not EC2Constants
        Show
        Andrei Savu added a comment - See https://github.com/jclouds/jclouds/blob/master/providers/aws-ec2/src/main/java/org/jclouds/aws/ec2/reference/AWSEC2Constants.java
        Hide
        Andrei Savu added a comment -

        I'm unable to get this to work as documented. If I set jclouds.ec.ami-query="" I get:

        Exception in thread "main" java.util.NoSuchElementException: imageId(us-east-1/ami-1b814f72) not found
        	at org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:567)
        	at org.apache.whirr.compute.BootstrapTemplate.build(BootstrapTemplate.java:75)
        	at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:100)
        	at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
        	at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
        	at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:62)
        	at org.apache.whirr.cli.Main.run(Main.java:64)
        	at org.apache.whirr.cli.Main.main(Main.java:97)
        

        Tried with Ubuntu 10.04 in us-east-1 by Canonical and Amazon Linux also in us-east-1.

        This is the change I've made in ClusterSpec:

           public void setImageId(String imageId) {
             this.imageId = imageId;
        +
        +    // Amazon specific optimization to avoid running queries across
        +    // all regions when the image-id is known from the property file
        +
        +    if ("aws-ec2".equals(getProvider())) {
        +      config.setProperty(AWSEC2Constants.PROPERTY_EC2_AMI_QUERY, "");
        +    }
           }
        
        Show
        Andrei Savu added a comment - I'm unable to get this to work as documented. If I set jclouds.ec.ami-query="" I get: Exception in thread "main" java.util.NoSuchElementException: imageId(us-east-1/ami-1b814f72) not found at org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:567) at org.apache.whirr.compute.BootstrapTemplate.build(BootstrapTemplate.java:75) at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:100) at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107) at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106) at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:62) at org.apache.whirr.cli.Main.run(Main.java:64) at org.apache.whirr.cli.Main.main(Main.java:97) Tried with Ubuntu 10.04 in us-east-1 by Canonical and Amazon Linux also in us-east-1. This is the change I've made in ClusterSpec: public void setImageId(String imageId) { this.imageId = imageId; + + // Amazon specific optimization to avoid running queries across + // all regions when the image-id is known from the property file + + if ("aws-ec2".equals(getProvider())) { + config.setProperty(AWSEC2Constants.PROPERTY_EC2_AMI_QUERY, ""); + } }
        Hide
        Andrei Savu added a comment -

        I also see this in the logs before getting the exception:

        2011-11-08 01:42:50,687 DEBUG [jclouds.compute] (main) >> searching params([biggest=false, fastest=false, imageName=null, imageDescription=null, imageId=us-east-1/ami-ad36fbc4, imagePredicate=null, imageVersion=null, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=1024, osFamily=null, osName=null, osDescription=null, osVersion=null, osArch=null, os64Bit=null, hardwareId=null])
        2011-11-08 01:42:50,687 DEBUG [jclouds.compute] (main) >> no jclouds.ec2.ami-query specified, skipping image parsing
        2011-11-08 01:42:50,689 DEBUG [jclouds.compute] (user thread 0) >> providing images
        2011-11-08 01:42:51,642 DEBUG [jclouds.compute] (user thread 0) << images(52)
        2011-11-08 01:42:51,668 DEBUG [jclouds.compute] (main) << images(52)
        

        I guess I'm missing something.

        Show
        Andrei Savu added a comment - I also see this in the logs before getting the exception: 2011-11-08 01:42:50,687 DEBUG [jclouds.compute] (main) >> searching params([biggest=false, fastest=false, imageName=null, imageDescription=null, imageId=us-east-1/ami-ad36fbc4, imagePredicate=null, imageVersion=null, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=1024, osFamily=null, osName=null, osDescription=null, osVersion=null, osArch=null, os64Bit=null, hardwareId=null]) 2011-11-08 01:42:50,687 DEBUG [jclouds.compute] (main) >> no jclouds.ec2.ami-query specified, skipping image parsing 2011-11-08 01:42:50,689 DEBUG [jclouds.compute] (user thread 0) >> providing images 2011-11-08 01:42:51,642 DEBUG [jclouds.compute] (user thread 0) << images(52) 2011-11-08 01:42:51,668 DEBUG [jclouds.compute] (main) << images(52) I guess I'm missing something.
        Hide
        Paul Baclace added a comment -

        Here are the relevant and coordinated properties I use to launch a private image in us-west-1 for aws-ec2 provider:

        whirr.provider=aws-ec2

        whirr.image-id=us-west-1/ami-ffffffff

        jclouds.ec2.ami-query=owner-id=999999999;state=available;image-type=machine;root-device-type=instance-store;architecture=x86_32

        jclouds.ec2.cc-regions=us-west-1

        jclouds.aws-s3.endpoint=us-west-1

        whirr.location-id=us-west-1

        whirr.hardware-id=c1.medium

        Show
        Paul Baclace added a comment - Here are the relevant and coordinated properties I use to launch a private image in us-west-1 for aws-ec2 provider: whirr.provider=aws-ec2 whirr.image-id=us-west-1/ami-ffffffff jclouds.ec2.ami-query=owner-id=999999999;state=available;image-type=machine;root-device-type=instance-store;architecture=x86_32 jclouds.ec2.cc-regions=us-west-1 jclouds.aws-s3.endpoint=us-west-1 whirr.location-id=us-west-1 whirr.hardware-id=c1.medium
        Hide
        Adrian Cole added a comment -

        small nits

        remove the line jclouds.ec2.cc-regions=us-west-1, as cc isn't available in us-west-1

        what you really want to do is say "don't parse cluster amis" which is this: "jclouds.ec2.cc-ami-query="

        moreover, if you always use explicit image ids, then set "jclouds.ec2.ami-query=" too

        I hope this helps!

        Show
        Adrian Cole added a comment - small nits remove the line jclouds.ec2.cc-regions=us-west-1, as cc isn't available in us-west-1 what you really want to do is say "don't parse cluster amis" which is this: "jclouds.ec2.cc-ami-query=" moreover, if you always use explicit image ids, then set "jclouds.ec2.ami-query=" too I hope this helps!
        Hide
        Andrei Savu added a comment -

        Finally got this to work. It seems like you need to set both jclouds.ec2.cc-ami-query and jclouds.ec2.ami-query to "". Patch in a minute.

        Show
        Andrei Savu added a comment - Finally got this to work. It seems like you need to set both jclouds.ec2.cc-ami-query and jclouds.ec2.ami-query to "". Patch in a minute.
        Hide
        Andrei Savu added a comment -

        Working patch tested on aws-ec2. The speed improvement is clearly visible. Ready for review.

        Show
        Andrei Savu added a comment - Working patch tested on aws-ec2. The speed improvement is clearly visible. Ready for review.
        Hide
        Andrei Savu added a comment -
        Show
        Andrei Savu added a comment - Now updating the jclouds docs in http://code.google.com/p/jclouds/issues/detail?id=748
        Hide
        Andrei Savu added a comment -

        Updated patch to address some offline comments from Adrian. Integration tests are working as expected.

        Show
        Andrei Savu added a comment - Updated patch to address some offline comments from Adrian. Integration tests are working as expected.
        Hide
        Andrei Savu added a comment -

        David? Adrian? What do you think?

        Show
        Andrei Savu added a comment - David? Adrian? What do you think?
        Hide
        Adrian Cole added a comment -

        +1 slick

        Show
        Adrian Cole added a comment - +1 slick
        Hide
        Andrei Savu added a comment -

        Committed. Thanks Adrian!

        Show
        Andrei Savu added a comment - Committed. Thanks Adrian!

          People

          • Assignee:
            Andrei Savu
            Reporter:
            Andrei Savu
          • Votes:
            3 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development