Apache Whirr (retired)
  1. Apache Whirr (retired)
  2. WHIRR-416

Enable lazy image fetching when the image-id is known

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: core
    • Labels:
      None

      Description

      "By default, the ComputeService will prefetch images from Alestic, Canonical, and RightScale. This allows you to query the images, which is great, if you don't know what you are looking for. However, if you already know the imageId you want, this is wasteful. We now provide the option to lazy-fetch images, which speed-ups execution."

      http://www.jclouds.org/documentation/userguide/using-ec2

      1. WHIRR-416.patch
        8 kB
        Andrei Savu
      2. WHIRR-416.patch
        7 kB
        Andrei Savu

        Activity

        Andrei Savu created issue -
        Andrei Savu made changes -
        Field Original Value New Value
        Assignee Andrei Savu [ savu.andrei ]
        Hide
        Andrei Savu added a comment -

        The jclouds documentation is a bit out of date, it should be AWSEC2Constants.PROPERTY_EC2_AMI_QUERY not EC2Constants

        Show
        Andrei Savu added a comment - The jclouds documentation is a bit out of date, it should be AWSEC2Constants.PROPERTY_EC2_AMI_QUERY not EC2Constants
        Show
        Andrei Savu added a comment - See https://github.com/jclouds/jclouds/blob/master/providers/aws-ec2/src/main/java/org/jclouds/aws/ec2/reference/AWSEC2Constants.java
        Hide
        Andrei Savu added a comment -

        I'm unable to get this to work as documented. If I set jclouds.ec.ami-query="" I get:

        Exception in thread "main" java.util.NoSuchElementException: imageId(us-east-1/ami-1b814f72) not found
        	at org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:567)
        	at org.apache.whirr.compute.BootstrapTemplate.build(BootstrapTemplate.java:75)
        	at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:100)
        	at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107)
        	at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
        	at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:62)
        	at org.apache.whirr.cli.Main.run(Main.java:64)
        	at org.apache.whirr.cli.Main.main(Main.java:97)
        

        Tried with Ubuntu 10.04 in us-east-1 by Canonical and Amazon Linux also in us-east-1.

        This is the change I've made in ClusterSpec:

           public void setImageId(String imageId) {
             this.imageId = imageId;
        +
        +    // Amazon specific optimization to avoid running queries across
        +    // all regions when the image-id is known from the property file
        +
        +    if ("aws-ec2".equals(getProvider())) {
        +      config.setProperty(AWSEC2Constants.PROPERTY_EC2_AMI_QUERY, "");
        +    }
           }
        
        Show
        Andrei Savu added a comment - I'm unable to get this to work as documented. If I set jclouds.ec.ami-query="" I get: Exception in thread "main" java.util.NoSuchElementException: imageId(us-east-1/ami-1b814f72) not found at org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:567) at org.apache.whirr.compute.BootstrapTemplate.build(BootstrapTemplate.java:75) at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:100) at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:107) at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106) at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:62) at org.apache.whirr.cli.Main.run(Main.java:64) at org.apache.whirr.cli.Main.main(Main.java:97) Tried with Ubuntu 10.04 in us-east-1 by Canonical and Amazon Linux also in us-east-1. This is the change I've made in ClusterSpec: public void setImageId(String imageId) { this.imageId = imageId; + + // Amazon specific optimization to avoid running queries across + // all regions when the image-id is known from the property file + + if ("aws-ec2".equals(getProvider())) { + config.setProperty(AWSEC2Constants.PROPERTY_EC2_AMI_QUERY, ""); + } }
        Hide
        Andrei Savu added a comment -

        I also see this in the logs before getting the exception:

        2011-11-08 01:42:50,687 DEBUG [jclouds.compute] (main) >> searching params([biggest=false, fastest=false, imageName=null, imageDescription=null, imageId=us-east-1/ami-ad36fbc4, imagePredicate=null, imageVersion=null, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=1024, osFamily=null, osName=null, osDescription=null, osVersion=null, osArch=null, os64Bit=null, hardwareId=null])
        2011-11-08 01:42:50,687 DEBUG [jclouds.compute] (main) >> no jclouds.ec2.ami-query specified, skipping image parsing
        2011-11-08 01:42:50,689 DEBUG [jclouds.compute] (user thread 0) >> providing images
        2011-11-08 01:42:51,642 DEBUG [jclouds.compute] (user thread 0) << images(52)
        2011-11-08 01:42:51,668 DEBUG [jclouds.compute] (main) << images(52)
        

        I guess I'm missing something.

        Show
        Andrei Savu added a comment - I also see this in the logs before getting the exception: 2011-11-08 01:42:50,687 DEBUG [jclouds.compute] (main) >> searching params([biggest=false, fastest=false, imageName=null, imageDescription=null, imageId=us-east-1/ami-ad36fbc4, imagePredicate=null, imageVersion=null, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=1024, osFamily=null, osName=null, osDescription=null, osVersion=null, osArch=null, os64Bit=null, hardwareId=null]) 2011-11-08 01:42:50,687 DEBUG [jclouds.compute] (main) >> no jclouds.ec2.ami-query specified, skipping image parsing 2011-11-08 01:42:50,689 DEBUG [jclouds.compute] (user thread 0) >> providing images 2011-11-08 01:42:51,642 DEBUG [jclouds.compute] (user thread 0) << images(52) 2011-11-08 01:42:51,668 DEBUG [jclouds.compute] (main) << images(52) I guess I'm missing something.
        Hide
        Paul Baclace added a comment -

        Here are the relevant and coordinated properties I use to launch a private image in us-west-1 for aws-ec2 provider:

        whirr.provider=aws-ec2

        whirr.image-id=us-west-1/ami-ffffffff

        jclouds.ec2.ami-query=owner-id=999999999;state=available;image-type=machine;root-device-type=instance-store;architecture=x86_32

        jclouds.ec2.cc-regions=us-west-1

        jclouds.aws-s3.endpoint=us-west-1

        whirr.location-id=us-west-1

        whirr.hardware-id=c1.medium

        Show
        Paul Baclace added a comment - Here are the relevant and coordinated properties I use to launch a private image in us-west-1 for aws-ec2 provider: whirr.provider=aws-ec2 whirr.image-id=us-west-1/ami-ffffffff jclouds.ec2.ami-query=owner-id=999999999;state=available;image-type=machine;root-device-type=instance-store;architecture=x86_32 jclouds.ec2.cc-regions=us-west-1 jclouds.aws-s3.endpoint=us-west-1 whirr.location-id=us-west-1 whirr.hardware-id=c1.medium
        Hide
        Adrian Cole (Inactive) added a comment -

        small nits

        remove the line jclouds.ec2.cc-regions=us-west-1, as cc isn't available in us-west-1

        what you really want to do is say "don't parse cluster amis" which is this: "jclouds.ec2.cc-ami-query="

        moreover, if you always use explicit image ids, then set "jclouds.ec2.ami-query=" too

        I hope this helps!

        Show
        Adrian Cole (Inactive) added a comment - small nits remove the line jclouds.ec2.cc-regions=us-west-1, as cc isn't available in us-west-1 what you really want to do is say "don't parse cluster amis" which is this: "jclouds.ec2.cc-ami-query=" moreover, if you always use explicit image ids, then set "jclouds.ec2.ami-query=" too I hope this helps!
        Hide
        Andrei Savu added a comment -

        Finally got this to work. It seems like you need to set both jclouds.ec2.cc-ami-query and jclouds.ec2.ami-query to "". Patch in a minute.

        Show
        Andrei Savu added a comment - Finally got this to work. It seems like you need to set both jclouds.ec2.cc-ami-query and jclouds.ec2.ami-query to "". Patch in a minute.
        Hide
        Andrei Savu added a comment -

        Working patch tested on aws-ec2. The speed improvement is clearly visible. Ready for review.

        Show
        Andrei Savu added a comment - Working patch tested on aws-ec2. The speed improvement is clearly visible. Ready for review.
        Andrei Savu made changes -
        Attachment WHIRR-416.patch [ 12503237 ]
        Andrei Savu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Andrei Savu added a comment -
        Show
        Andrei Savu added a comment - Now updating the jclouds docs in http://code.google.com/p/jclouds/issues/detail?id=748
        Hide
        Andrei Savu added a comment -

        Updated patch to address some offline comments from Adrian. Integration tests are working as expected.

        Show
        Andrei Savu added a comment - Updated patch to address some offline comments from Adrian. Integration tests are working as expected.
        Andrei Savu made changes -
        Attachment WHIRR-416.patch [ 12503429 ]
        Hide
        Andrei Savu added a comment -

        David? Adrian? What do you think?

        Show
        Andrei Savu added a comment - David? Adrian? What do you think?
        Hide
        Adrian Cole (Inactive) added a comment -

        +1 slick

        Show
        Adrian Cole (Inactive) added a comment - +1 slick
        Hide
        Andrei Savu added a comment -

        Committed. Thanks Adrian!

        Show
        Andrei Savu added a comment - Committed. Thanks Adrian!
        Andrei Savu made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        12d 2h 46m 1 Andrei Savu 10/Nov/11 15:22
        Patch Available Patch Available Resolved Resolved
        4d 4h 16m 1 Andrei Savu 14/Nov/11 19:38

          People

          • Assignee:
            Andrei Savu
            Reporter:
            Andrei Savu
          • Votes:
            3 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development