Whirr
  1. Whirr
  2. WHIRR-182

Support deployment to Eucalyptus clouds

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: new provider
    • Labels:
      None

      Description

      Eucalyptus is supported in jclouds as the "eucalyptus" provider [1]. Adding support should be fairly straightforward.

      [0] http://www.eucalyptus.com/
      [1] http://code.google.com/p/jclouds/wiki/QuickStartEucalyptus

        Activity

        Hide
        Andrei Savu added a comment -

        I no longer have an environment and the time needed to work on this issue. It would be great if you guys at Eucalyptus could provide a patch.

        Show
        Andrei Savu added a comment - I no longer have an environment and the time needed to work on this issue. It would be great if you guys at Eucalyptus could provide a patch.
        Hide
        Sang-Min Park added a comment -

        Hello,
        Any progress on the patch? I'm an engineer at Eucalyptus and I'd be happy to discuss anything related to Eucalyptus.
        Cheers,

        Sang-min

        Show
        Sang-Min Park added a comment - Hello, Any progress on the patch? I'm an engineer at Eucalyptus and I'd be happy to discuss anything related to Eucalyptus. Cheers, Sang-min
        Hide
        Anirban Mandal added a comment -

        Thank you for the update. Please post when your patch is available.

        Regards,

        • Anirban
        Show
        Anirban Mandal added a comment - Thank you for the update. Please post when your patch is available. Regards, Anirban
        Hide
        Andrei Savu added a comment -

        Currently Whirr does not support Eucalyptus based clouds. I'm working on a patch for this issue. I'm have installed Eucalyptus on a local server in SYSTEM mode and I've been able to start virtual machines using Whirr. I'm still having some authentication related issues and I still need to test the patch using Eucalyptus in MANAGED* mode. I will post a version of the patch I'm working on soon.

        Show
        Andrei Savu added a comment - Currently Whirr does not support Eucalyptus based clouds. I'm working on a patch for this issue. I'm have installed Eucalyptus on a local server in SYSTEM mode and I've been able to start virtual machines using Whirr. I'm still having some authentication related issues and I still need to test the patch using Eucalyptus in MANAGED* mode. I will post a version of the patch I'm working on soon.
        Hide
        Anirban Mandal added a comment -

        Hi,

        I was trying to test whirr with Eucalyptus clouds . I got a Eucalyptus community cloud account and tried to launch a hadoop cluster using whirr CLI. I am using 0.4.0-incubating-SNAPSHOT version of whirr.

        I have two issues:

        1. How do I specify an eucalyptus endpoint ? I understand that a property called 'eucalyptus.endpoint' needs to be set for jclouds to be able to talk to a eucalyptus cloud. But, I didn't see how I can configure this property in whirr. Somewhere internally, whirr knows to talk to the Eucalyptus community cloud at http://173.205.188.130:8773/services/Eucalyptus/

        I hacked up core/src/main/java/org/apache/whirr/service/ComputeServiceContextBuilder.java by setting eucalyptus.endpoint and passing an updated property to createContext() in order to test on our own Eucalyptus cluster. But, that is definitely not the way to go forward. Please suggest.

        2. The second issue is of bigger concern. For both the Eucalyptus clouds I tested this on, I see that that the instances remain in pending state and after a while gets terminated. The last things I see in the whirr log files is

        ....
        2011-03-02 14:50:11,121 DEBUG [jclouds.compute] (main) << matched image([id=Eucalyptus/emi-9ACB1363, name=null, operatingSystem=[name=null, family=centos, version=, arch=paravirtual, is64Bit=true, description=centos-5.3-x86_64/centos.5-3.x86-64.img.manifest.xml], description=centos-5.3-x86_64/centos.5-3.x86-64.img.manifest.xml, version=null, location=[id=Eucalyptus, scope=REGION, description=Eucalyptus, parent=eucalyptus, iso3166Codes=[], metadata={}], loginUser=root, userMetadata=

        {owner=admin, rootDeviceType=instance-store}

        ])2011-03-02 14:50:11,122 INFO [org.apache.whirr.cluster.actions.NodeStarter] (pool-1-thread-4) Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode]2011-03-02 14:50:11,122 DEBUG [jclouds.compute] (pool-1-thread-4) >> running 1 node group(myhadoopcluster) location(Eucalyptus) image(Eucalyptus/emi-9ACB1363) hardwareProfile(m1.small) options([groupIds=[], keyPair=null, noKeyPair=false, userData=null, blockDeviceMappings=[]])
        2011-03-02 14:50:11,122 DEBUG [jclouds.compute] (pool-1-thread-4) >> creating keyPair region(Eucalyptus) tag(myhadoopcluster)
        2011-03-02 14:50:11,879 DEBUG [jclouds.compute] (pool-1-thread-2) << created keyPair(jclouds#myhadoopcluster#Eucalyptus#93)2011-03-02 14:50:11,880 DEBUG [jclouds.compute] (pool-1-thread-2) >> creating securityGroup region(Eucalyptus) name(jclouds#myhadoopcluster#Eucalyptus)
        2011-03-02 14:50:12,289 DEBUG [jclouds.compute] (pool-1-thread-2) << reused securityGroup(jclouds#myhadoopcluster#Eucaly
        ptus)
        2011-03-02 14:50:12,290 DEBUG [jclouds.compute] (pool-1-thread-2) >> running 1 instance region(Eucalyptus) zone(null) am
        i(emi-9ACB1363) params(

        {InstanceType=[m1.small], SecurityGroup.1=[jclouds#myhadoopcluster#Eucalyptus], KeyName=[jclouds# myhadoopcluster#Eucalyptus#93]}

        )
        ....
        2011-03-02 14:50:12,998 DEBUG [jclouds.compute] (pool-1-thread-2) << started instances(i-51020A57)
        2011-03-02 14:50:13,041 DEBUG [jclouds.compute] (pool-1-thread-4) << started instances(i-4A010867)
        2011-03-02 14:50:13,157 DEBUG [jclouds.compute] (pool-1-thread-2) << present instances(i-51020A57)
        2011-03-02 14:50:13,191 DEBUG [jclouds.compute] (pool-1-thread-4) << present instances(i-4A010867)
        ... nothing after that

        When we examine the logs on the Eucalyptus Cloud head node, we see that the neither the cloud controller nor the node controllers have kicked in. The request to Eucalyptus has died in the cloud front end itself. So, the pending state has nothing to do with faulty images, if any. The exceptions on the cloud-debug.log mostly look like

        16:19:03 ERROR [RestfulMarshallingHandler:ReplyQueue.16] java.lang.NullPointerException
        java.lang.NullPointerException
        ...

        , which might mean that the jclouds clients are not being able to talk to the Euca cloud . But, I have no idea beyond that.

        Am I not using whirr correctly for Eucalyptus ? The properties file I use looks like:

        whirr.cluster-name=myhadoopcluster
        whirr.instance-templates=1 jt+nn,1 dn+tt
        whirr.provider=eucalyptus
        whirr.identity=<ACCESS>
        whirr.credential=<SECRET>
        whirr.private-key-file=/home/orca/.ssh/id_rsa
        whirr.public-key-file=/home/orca/.ssh/id_rsa.pub
        whirr.image-id=Eucalyptus/emi-9ACB1363

        Regards,

        • Anirban
        Show
        Anirban Mandal added a comment - Hi, I was trying to test whirr with Eucalyptus clouds . I got a Eucalyptus community cloud account and tried to launch a hadoop cluster using whirr CLI. I am using 0.4.0-incubating-SNAPSHOT version of whirr. I have two issues: 1. How do I specify an eucalyptus endpoint ? I understand that a property called 'eucalyptus.endpoint' needs to be set for jclouds to be able to talk to a eucalyptus cloud. But, I didn't see how I can configure this property in whirr. Somewhere internally, whirr knows to talk to the Eucalyptus community cloud at http://173.205.188.130:8773/services/Eucalyptus/ I hacked up core/src/main/java/org/apache/whirr/service/ComputeServiceContextBuilder.java by setting eucalyptus.endpoint and passing an updated property to createContext() in order to test on our own Eucalyptus cluster. But, that is definitely not the way to go forward. Please suggest. 2. The second issue is of bigger concern. For both the Eucalyptus clouds I tested this on, I see that that the instances remain in pending state and after a while gets terminated. The last things I see in the whirr log files is .... 2011-03-02 14:50:11,121 DEBUG [jclouds.compute] (main) << matched image([id=Eucalyptus/emi-9ACB1363, name=null, operatingSystem= [name=null, family=centos, version=, arch=paravirtual, is64Bit=true, description=centos-5.3-x86_64/centos.5-3.x86-64.img.manifest.xml] , description=centos-5.3-x86_64/centos.5-3.x86-64.img.manifest.xml, version=null, location=[id=Eucalyptus, scope=REGION, description=Eucalyptus, parent=eucalyptus, iso3166Codes=[], metadata={}], loginUser=root, userMetadata= {owner=admin, rootDeviceType=instance-store} ])2011-03-02 14:50:11,122 INFO [org.apache.whirr.cluster.actions.NodeStarter] (pool-1-thread-4) Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode] 2011-03-02 14:50:11,122 DEBUG [jclouds.compute] (pool-1-thread-4) >> running 1 node group(myhadoopcluster) location(Eucalyptus) image(Eucalyptus/emi-9ACB1363) hardwareProfile(m1.small) options([groupIds=[], keyPair=null, noKeyPair=false, userData=null, blockDeviceMappings=[]]) 2011-03-02 14:50:11,122 DEBUG [jclouds.compute] (pool-1-thread-4) >> creating keyPair region(Eucalyptus) tag(myhadoopcluster) 2011-03-02 14:50:11,879 DEBUG [jclouds.compute] (pool-1-thread-2) << created keyPair(jclouds#myhadoopcluster#Eucalyptus#93)2011-03-02 14:50:11,880 DEBUG [jclouds.compute] (pool-1-thread-2) >> creating securityGroup region(Eucalyptus) name(jclouds#myhadoopcluster#Eucalyptus) 2011-03-02 14:50:12,289 DEBUG [jclouds.compute] (pool-1-thread-2) << reused securityGroup(jclouds#myhadoopcluster#Eucaly ptus) 2011-03-02 14:50:12,290 DEBUG [jclouds.compute] (pool-1-thread-2) >> running 1 instance region(Eucalyptus) zone(null) am i(emi-9ACB1363) params( {InstanceType=[m1.small], SecurityGroup.1=[jclouds#myhadoopcluster#Eucalyptus], KeyName=[jclouds# myhadoopcluster#Eucalyptus#93]} ) .... 2011-03-02 14:50:12,998 DEBUG [jclouds.compute] (pool-1-thread-2) << started instances(i-51020A57) 2011-03-02 14:50:13,041 DEBUG [jclouds.compute] (pool-1-thread-4) << started instances(i-4A010867) 2011-03-02 14:50:13,157 DEBUG [jclouds.compute] (pool-1-thread-2) << present instances(i-51020A57) 2011-03-02 14:50:13,191 DEBUG [jclouds.compute] (pool-1-thread-4) << present instances(i-4A010867) ... nothing after that When we examine the logs on the Eucalyptus Cloud head node, we see that the neither the cloud controller nor the node controllers have kicked in. The request to Eucalyptus has died in the cloud front end itself. So, the pending state has nothing to do with faulty images, if any. The exceptions on the cloud-debug.log mostly look like 16:19:03 ERROR [RestfulMarshallingHandler:ReplyQueue.16] java.lang.NullPointerException java.lang.NullPointerException ... , which might mean that the jclouds clients are not being able to talk to the Euca cloud . But, I have no idea beyond that. Am I not using whirr correctly for Eucalyptus ? The properties file I use looks like: whirr.cluster-name=myhadoopcluster whirr.instance-templates=1 jt+nn,1 dn+tt whirr.provider=eucalyptus whirr.identity=<ACCESS> whirr.credential=<SECRET> whirr.private-key-file=/home/orca/.ssh/id_rsa whirr.public-key-file=/home/orca/.ssh/id_rsa.pub whirr.image-id=Eucalyptus/emi-9ACB1363 Regards, Anirban
        Hide
        Andrei Savu added a comment -

        Thanks Adrian for letting me know. I will give it a try.

        Show
        Andrei Savu added a comment - Thanks Adrian for letting me know. I will give it a try.
        Hide
        Adrian Cole added a comment -

        note that if you use the provider "eucalyptus" you will need to set the property "eucalyptus.endpoint" = "http://yourhost:8773/services/Eucalyptus" when creating the ComputeServiceContext

        Show
        Adrian Cole added a comment - note that if you use the provider "eucalyptus" you will need to set the property "eucalyptus.endpoint" = "http://yourhost:8773/services/Eucalyptus" when creating the ComputeServiceContext
        Hide
        Adrian Cole added a comment -

        jclouds for eucalyptus now works in snapshot. if you'd like to test, use the following information:

        maven dependency: org.jclouds.api/eucalyptus
        ComputeService provider: "eucalyptus"

        if you have a partnercloud account, you can use:
        maven dependency: org.jclouds.provider/eucalyptus-partnercloud-ec2
        ComputeService provider: "eucalyptus-partnercloud-ec2"

        note that the snapshot repo is: https://oss.sonatype.org/content/repositories/snapshots

        This support is finalized and will be included in the pending jclouds 1.0-beta-9 release

        Show
        Adrian Cole added a comment - jclouds for eucalyptus now works in snapshot. if you'd like to test, use the following information: maven dependency: org.jclouds.api/eucalyptus ComputeService provider: "eucalyptus" if you have a partnercloud account, you can use: maven dependency: org.jclouds.provider/eucalyptus-partnercloud-ec2 ComputeService provider: "eucalyptus-partnercloud-ec2" note that the snapshot repo is: https://oss.sonatype.org/content/repositories/snapshots This support is finalized and will be included in the pending jclouds 1.0-beta-9 release
        Hide
        Andrei Savu added a comment -

        I now have access to the ECC Eucalyptus cloud. I'm starting to work on a patch.

        Show
        Andrei Savu added a comment - I now have access to the ECC Eucalyptus cloud. I'm starting to work on a patch.
        Hide
        Adrian Cole added a comment -

        expect a couple glitches until the latest version of eucalyptus is released. I've been working with David from Eucalyptus, and we should be ready to integration test again within a week, hopefully with all passing on jclouds-1.0-SNAPSHOT. The eucalyptus-specific fixes in jclouds snapshot could be patched inside whirr once all is verified.

        Show
        Adrian Cole added a comment - expect a couple glitches until the latest version of eucalyptus is released. I've been working with David from Eucalyptus, and we should be ready to integration test again within a week, hopefully with all passing on jclouds-1.0-SNAPSHOT. The eucalyptus-specific fixes in jclouds snapshot could be patched inside whirr once all is verified.
        Hide
        Andrei Savu added a comment -

        I have signed up for an account on http://ecc.eucalyptus.com/ and I have got a response from Steven Fitzgerald. I should be able to use that infrastructure for running the integration tests after I send the signed contributor agreement. I was planning to deploy a local private cloud.

        Show
        Andrei Savu added a comment - I have signed up for an account on http://ecc.eucalyptus.com/ and I have got a response from Steven Fitzgerald. I should be able to use that infrastructure for running the integration tests after I send the signed contributor agreement. I was planning to deploy a local private cloud.
        Hide
        Andrei Savu added a comment -

        Side note: It would be great if the Apache Foundation could provide a Eucalyptus based private cloud (maybe using Ubuntu) that could be used by a CI server for running the integration tests.

        Show
        Andrei Savu added a comment - Side note: It would be great if the Apache Foundation could provide a Eucalyptus based private cloud (maybe using Ubuntu) that could be used by a CI server for running the integration tests.

          People

          • Assignee:
            Unassigned
            Reporter:
            Andrei Savu
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development