Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.1
    • Component/s: None
    • Labels:
      None

      Description

      Jclouds 1.5.0 brings a lot of improvements and a set of new providers we can use.

      Adrian added on the discussion list:

      "Jclouds 1.5 has a concept called TemplateBuilderSpec which allows you to
      declare a String representation of TemplateBuilder mechanics.

      This allows you to set a property to override the default templates used in
      jclouds.

      For ex.

      aws-ec2.template=imageId=ami-foo,hardwareId=m1.large,locationId=eu-west-1

      Importantly, these are just strings, so you can publish a series per
      operating system family, and choose at runtime with
      templateBuilder.from(spec)

      Ex. You can make your own Map<String, TemplateBuilderSpec> with your
      favorite choices, perhaps locking in templates by id so you don't have to
      maintain the brittle java install scripts.

      Ex.

      templates.get("oneiric-java-ec2");

      Regardless of how this is used, the important part is that the constant
      complaints of automatic selection can now be over, as we can provide a
      properties syntax to afford users a means to supply their preferences
      declaratively."

      1. WHIRR-593_beta10.patch
        3 kB
        Adrian Cole
      2. WHIRR-593.patch
        39 kB
        Adrian Cole
      3. WHIRR-593+fix-zookeeper-test.patch
        46 kB
        Cosmin Dumitrache
      4. WHIRR-593-beta.11.patch
        0.5 kB
        Andrew Bayer
      5. WHIRR-593-incTemplate.patch
        50 kB
        Adrian Cole
      6. WHIRR-593-incTemplate.patch
        49 kB
        Adrian Cole
      7. WHIRR-593-mk4.patch
        46 kB
        Andrew Bayer
      8. WHIRR-593-mk5.patch
        45 kB
        Andrew Bayer
      9. WHIRR-593-sec-groups.patch
        2 kB
        Andrew Bayer

        Issue Links

          Activity

          Hide
          Adrian Cole (Inactive) added a comment -

          1.4.0 is not a stable release compared to 1.5. I'll help make a patch, especially as I have a demo in a couple days at uberconf!!

          Show
          Adrian Cole (Inactive) added a comment - 1.4.0 is not a stable release compared to 1.5. I'll help make a patch, especially as I have a demo in a couple days at uberconf!!
          Hide
          Adrian Cole (Inactive) added a comment -

          Please help test this patch!

          Note that jclouds 1.5 gives the following new providers and apis, so probably worth your while to help

          cloudstack (version 3.03)
          greenqloud-compute (carbon neutral service)
          joyentcloud (smartos)
          openstack-nova (essex)

          Show
          Adrian Cole (Inactive) added a comment - Please help test this patch! Note that jclouds 1.5 gives the following new providers and apis, so probably worth your while to help cloudstack (version 3.03) greenqloud-compute (carbon neutral service) joyentcloud (smartos) openstack-nova (essex)
          Hide
          Andrei Savu added a comment -

          Builds fine but I get a failure when running the ZooKeeper itests. I am investigating.

          Show
          Andrei Savu added a comment - Builds fine but I get a failure when running the ZooKeeper itests. I am investigating.
          Hide
          Andrew Bayer added a comment -

          fwiw, I'm getting a failure in the Cassandra itests, but I'm getting that with 1.4 or 1.5.

          Show
          Andrew Bayer added a comment - fwiw, I'm getting a failure in the Cassandra itests, but I'm getting that with 1.4 or 1.5.
          Hide
          Adrian Cole (Inactive) added a comment -

          p.s. I used the following with zookeeper on version 0.7 today. Otherwise, it didn't work (ex. crash/burn)

          --image-id=us-west-1/ami-33fda576 --location-id=us-west-1

          Show
          Adrian Cole (Inactive) added a comment - p.s. I used the following with zookeeper on version 0.7 today. Otherwise, it didn't work (ex. crash/burn) --image-id=us-west-1/ami-33fda576 --location-id=us-west-1
          Hide
          Andrew Bayer added a comment -

          I just verified ZK working on trunk with 1.4, trying now with 1.5...

          Show
          Andrew Bayer added a comment - I just verified ZK working on trunk with 1.4, trying now with 1.5...
          Hide
          Andrew Bayer added a comment -

          And with 1.5, it's hanging. Hopefully it'll time out and tell me why it's hanging at some point. =)

          Show
          Andrew Bayer added a comment - And with 1.5, it's hanging. Hopefully it'll time out and tell me why it's hanging at some point. =)
          Hide
          Andrei Savu added a comment -

          I've just tried the itests from core - they fail on cloudservers-us but work file on aws-ec2.

          Show
          Andrei Savu added a comment - I've just tried the itests from core - they fail on cloudservers-us but work file on aws-ec2.
          Hide
          Cosmin Dumitrache added a comment - - edited

          Here's a new version of the patch that fixes the failing ZooKeeper integration tests.

          This worked for me when testing against AWS.

          It's changing the naming convention used for some environment variables and removes the region name from the security group name.

          Show
          Cosmin Dumitrache added a comment - - edited Here's a new version of the patch that fixes the failing ZooKeeper integration tests. This worked for me when testing against AWS. It's changing the naming convention used for some environment variables and removes the region name from the security group name.
          Hide
          Andrei Savu added a comment -

          Reviewing now. I will try the Hadoop itests.

          Show
          Andrei Savu added a comment - Reviewing now. I will try the Hadoop itests.
          Hide
          Andrei Savu added a comment -

          Hadoop itests work fine aws-ec2. Still testing on cloudservers-us.

          Show
          Andrei Savu added a comment - Hadoop itests work fine aws-ec2. Still testing on cloudservers-us.
          Hide
          Andrei Savu added a comment -

          On cloudservers-us it fails with "Exhausted available authentication methods". Any ideas on what's different from jclouds 1.4.0?

          Show
          Andrei Savu added a comment - On cloudservers-us it fails with "Exhausted available authentication methods". Any ideas on what's different from jclouds 1.4.0?
          Hide
          Adrian Cole (Inactive) added a comment -

          not sure. I'll run jclouds regression test on cloudservers-us first and see if all ok

          Show
          Adrian Cole (Inactive) added a comment - not sure. I'll run jclouds regression test on cloudservers-us first and see if all ok
          Hide
          Cosmin Dumitrache added a comment -

          Here's a quick summary with some of the stuff I've been trying in the hopes that it might be useful for people:

          • PhaseExecutionBarrierTest passes on AWS but no VM ever shows up in the AWS web console. I tried this in different regions and the result is the same. Here's the test output.
          • Running this on rackspace produces "Exhausted available authentication methods".
            Here's the test output.
          • Was able to tell it to use my ssh key from ~/.ssh as opposed to the random one but still no luck. Looking in the debugger, it seems like it's using the correct key. Also, I was able to ssh to the rackspace machine after the test failed so this kind of indicates that the key it was using was indeed correct even though it's complaining.
          • Not having a catch clause here seems a bit worrying but testing showed that no exception was being thrown anyway.
          Show
          Cosmin Dumitrache added a comment - Here's a quick summary with some of the stuff I've been trying in the hopes that it might be useful for people: PhaseExecutionBarrierTest passes on AWS but no VM ever shows up in the AWS web console. I tried this in different regions and the result is the same. Here's the test output . Running this on rackspace produces "Exhausted available authentication methods". Here's the test output . Was able to tell it to use my ssh key from ~/.ssh as opposed to the random one but still no luck. Looking in the debugger, it seems like it's using the correct key. Also, I was able to ssh to the rackspace machine after the test failed so this kind of indicates that the key it was using was indeed correct even though it's complaining. Not having a catch clause here seems a bit worrying but testing showed that no exception was being thrown anyway.
          Hide
          Cosmin Dumitrache added a comment -

          Here's my jclouds ssh log for running PhaseExecutionBarrierTest on Rackspace in case someone finds it helpful.
          I see that it's ssh-ing in as root for the boostrap phase (that works fine), but for the configure phase it's ssh-ing in as my user (and fails). As I said before, it seems odd because I'm able to manually ssh as my user.

          Show
          Cosmin Dumitrache added a comment - Here's my jclouds ssh log for running PhaseExecutionBarrierTest on Rackspace in case someone finds it helpful. I see that it's ssh-ing in as root for the boostrap phase (that works fine), but for the configure phase it's ssh-ing in as my user (and fails). As I said before, it seems odd because I'm able to manually ssh as my user.
          Hide
          Andrew Bayer added a comment -

          fyi, I'm able to run hadoop and zk itests on CloudStack 3.0.3 (with a few tweaks I needed to deal with our CS setup being a bit strange).

          Show
          Andrew Bayer added a comment - fyi, I'm able to run hadoop and zk itests on CloudStack 3.0.3 (with a few tweaks I needed to deal with our CS setup being a bit strange).
          Hide
          Andrew Bayer added a comment -

          Found an additional issue with security group naming - 1.5 doesn't append "#region" to EC2 security group names, so FirewallManager needed to be updated. This patch would go on top of the existing one.

          Show
          Andrew Bayer added a comment - Found an additional issue with security group naming - 1.5 doesn't append "#region" to EC2 security group names, so FirewallManager needed to be updated. This patch would go on top of the existing one.
          Hide
          Andrew Bayer added a comment -

          So what more do we need to do to get this in? The only problem I've still been seeing is that weird "exhausted available authentication methods" one.

          Also, I see that Cosmin's patch (which for some reason I hadn't already applied) deals with FirewallManager's #(region) name, but not FirewallManagerTest - my patch did them both, but I'll just grab Cosmin's patch and add FirewallManagerTest to it.

          Show
          Andrew Bayer added a comment - So what more do we need to do to get this in? The only problem I've still been seeing is that weird "exhausted available authentication methods" one. Also, I see that Cosmin's patch (which for some reason I hadn't already applied) deals with FirewallManager's #(region) name, but not FirewallManagerTest - my patch did them both, but I'll just grab Cosmin's patch and add FirewallManagerTest to it.
          Hide
          Andrei Savu added a comment -

          We need to find the root cause for those "exhausted available authentication methods" on cloudservers-* before committing.

          Show
          Andrei Savu added a comment - We need to find the root cause for those "exhausted available authentication methods" on cloudservers-* before committing.
          Hide
          Andrew Bayer added a comment -

          fwiw, it looked to me like Whirr was for some reason trying to use password authentication. It's claiming the rsa/sha1 fingerprints in jclouds-ssh.log, but the actual stacktrace refers to authPassword.

          Show
          Andrew Bayer added a comment - fwiw, it looked to me like Whirr was for some reason trying to use password authentication. It's claiming the rsa/sha1 fingerprints in jclouds-ssh.log, but the actual stacktrace refers to authPassword.
          Hide
          Andrew Bayer added a comment -

          Hmm - I'm not actually seeing any test failures in services/hadoop or services/zookeeper against jclouds 1.5. Which specific tests are failing?

          Show
          Andrew Bayer added a comment - Hmm - I'm not actually seeing any test failures in services/hadoop or services/zookeeper against jclouds 1.5. Which specific tests are failing?
          Hide
          Andrei Savu added a comment -

          I will try again later today and report back.

          Show
          Andrei Savu added a comment - I will try again later today and report back.
          Hide
          Andrew Bayer added a comment -

          Hmmmm - I tried adding ".password(null)" to the LoginCredentials calls without it, in ClusterController, and that seems to have fixed the PhaseExecutionBarrierTest issues on cloudservers-us. Double-checking now.

          Show
          Andrew Bayer added a comment - Hmmmm - I tried adding ".password(null)" to the LoginCredentials calls without it, in ClusterController, and that seems to have fixed the PhaseExecutionBarrierTest issues on cloudservers-us. Double-checking now.
          Hide
          Andrew Bayer added a comment -

          Ah, and I'd already tried adding password(null) to ScriptBasedClusterAction. Seems like something is keeping a non-null password value there for cases where there was at some point a password - e.g., cloudservers/cloudstack.

          Show
          Andrew Bayer added a comment - Ah, and I'd already tried adding password(null) to ScriptBasedClusterAction. Seems like something is keeping a non-null password value there for cases where there was at some point a password - e.g., cloudservers/cloudstack.
          Hide
          Andrew Bayer added a comment -

          And fyi, I'm able to run against rackspace-cloudservers-us just as well as against anything else, so we've got openstack support. =)

          Show
          Andrew Bayer added a comment - And fyi, I'm able to run against rackspace-cloudservers-us just as well as against anything else, so we've got openstack support. =)
          Hide
          Andrew Bayer added a comment -

          Latest patch (WHIRR-593-mk4.patch) incorporates everything so far. I switched to .noPassword() instead of .password(null), since that seemed more correct. I'm more wary on rackspace next-gen than I was earlier - it seems to still have some issues with launching/tearing down instances in some contexts, but I'm not sure if that's just my account or what.

          Otherwise, PhaseExecutionBarrierTest passes on cloudservers-us and aws-ec2. Hadoop and ZK tests pass on both as well.

          Show
          Andrew Bayer added a comment - Latest patch ( WHIRR-593 -mk4.patch) incorporates everything so far. I switched to .noPassword() instead of .password(null), since that seemed more correct. I'm more wary on rackspace next-gen than I was earlier - it seems to still have some issues with launching/tearing down instances in some contexts, but I'm not sure if that's just my account or what. Otherwise, PhaseExecutionBarrierTest passes on cloudservers-us and aws-ec2. Hadoop and ZK tests pass on both as well.
          Hide
          Adrian Cole (Inactive) added a comment -

          can you try removing the .noPassword() and using jclouds 1.5.0-SNAPSHOT? I think the following commit should sort that:
          https://github.com/jclouds/jclouds/commit/9d007a40b726bb3de492a947d83b98ddced872a5

          If that works, I can cut 1.5.0-beta.8 tomorrow.

          Show
          Adrian Cole (Inactive) added a comment - can you try removing the .noPassword() and using jclouds 1.5.0-SNAPSHOT? I think the following commit should sort that: https://github.com/jclouds/jclouds/commit/9d007a40b726bb3de492a947d83b98ddced872a5 If that works, I can cut 1.5.0-beta.8 tomorrow.
          Hide
          Andrew Bayer added a comment -

          New patch rolling out the noPassword() change and bumping the jclouds dependency to the as-yet-unreleased-but-coming-today 1.5.0-beta.8, which has the needed fix on the jclouds side.

          Show
          Andrew Bayer added a comment - New patch rolling out the noPassword() change and bumping the jclouds dependency to the as-yet-unreleased-but-coming-today 1.5.0-beta.8, which has the needed fix on the jclouds side.
          Hide
          Tom White added a comment -

          This looks good to me. +1 pending beta-8.

          Show
          Tom White added a comment - This looks good to me. +1 pending beta-8.
          Hide
          Adrian Cole (Inactive) added a comment - - edited

          tested with jclouds 1.5.0-beta.8 recently synced to maven central.

          committed WHIRR-593-mk5.patch as revision 1365445

          Thanks, Andrei, Cosmin & Andrew for writing this. Thanks, Tom for checking!

          Show
          Adrian Cole (Inactive) added a comment - - edited tested with jclouds 1.5.0-beta.8 recently synced to maven central. committed WHIRR-593 -mk5.patch as revision 1365445 Thanks, Andrei, Cosmin & Andrew for writing this. Thanks, Tom for checking!
          Hide
          Andrei Savu added a comment -

          Marking this as fixed. Thanks guys!

          Show
          Andrei Savu added a comment - Marking this as fixed. Thanks guys!
          Hide
          Adrian Cole (Inactive) added a comment -

          jclouds 1.5 isn't released, yet, so let's keep this open to track any updates based on betas, release candidates, etc

          Show
          Adrian Cole (Inactive) added a comment - jclouds 1.5 isn't released, yet, so let's keep this open to track any updates based on betas, release candidates, etc
          Hide
          Adrian Cole (Inactive) added a comment -

          bumped to jclouds 1.5.0-beta.9
          Committed revision 1369253.

          Show
          Adrian Cole (Inactive) added a comment - bumped to jclouds 1.5.0-beta.9 Committed revision 1369253.
          Hide
          Adrian Cole (Inactive) added a comment -

          For use with the following test repo. Please report back to http://code.google.com/p/jclouds/issues/detail?id=1068 when done!

          <repository>
          <id>jclouds-staging</id>
          <url>https://oss.sonatype.org/content/repositories/orgjclouds-039</url>
          <releases>
          <enabled>true</enabled>
          </releases>
          <snapshots>
          <enabled>false</enabled>
          </snapshots>
          </repository>

          Show
          Adrian Cole (Inactive) added a comment - For use with the following test repo. Please report back to http://code.google.com/p/jclouds/issues/detail?id=1068 when done! <repository> <id>jclouds-staging</id> <url> https://oss.sonatype.org/content/repositories/orgjclouds-039 </url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </repository>
          Hide
          Adrian Cole (Inactive) added a comment -

          The following patch adds whirr.template parameter which can override all the other image, hardware, location, or login-user parameters

          Note, I also implemented it for the undocumented group-based parameters as templates.group.template

          Show
          Adrian Cole (Inactive) added a comment - The following patch adds whirr.template parameter which can override all the other image, hardware, location, or login-user parameters Note, I also implemented it for the undocumented group-based parameters as templates.group.template
          Hide
          Adrian Cole (Inactive) added a comment -

          I'm recutting this patch. 10m

          Show
          Adrian Cole (Inactive) added a comment - I'm recutting this patch. 10m
          Hide
          Adrian Cole (Inactive) added a comment -

          replacement for previous patch

          Show
          Adrian Cole (Inactive) added a comment - replacement for previous patch
          Hide
          Adrian Cole (Inactive) added a comment -

          after applying this patch, please execute:
          svn rm --force ./core/src/main/java/org/apache/whirr/service/jclouds/TemplateBuilderStrategy.java ./core/src/test/java/org/apache/whirr/service/TemplateBuilderStrategyTest.java

          Show
          Adrian Cole (Inactive) added a comment - after applying this patch, please execute: svn rm --force ./core/src/main/java/org/apache/whirr/service/jclouds/TemplateBuilderStrategy.java ./core/src/test/java/org/apache/whirr/service/TemplateBuilderStrategyTest.java
          Hide
          Adrian Cole (Inactive) added a comment -

          updated to fix paul's error

          Show
          Adrian Cole (Inactive) added a comment - updated to fix paul's error
          Hide
          Adrian Cole (Inactive) added a comment -

          updated with spot price fix and tests

          Show
          Adrian Cole (Inactive) added a comment - updated with spot price fix and tests
          Hide
          Andrew Bayer added a comment -

          +1

          Show
          Andrew Bayer added a comment - +1
          Hide
          Adrian Cole (Inactive) added a comment -

          1.5.0-beta.10 in now
          Committed revision 1374393.

          Show
          Adrian Cole (Inactive) added a comment - 1.5.0-beta.10 in now Committed revision 1374393.
          Hide
          Andrew Bayer added a comment -

          tiny patch to bump to jclouds 1.5.0-beta.11 now that it's in Maven Central.

          Show
          Andrew Bayer added a comment - tiny patch to bump to jclouds 1.5.0-beta.11 now that it's in Maven Central.
          Hide
          Adrian Cole (Inactive) added a comment -

          +1

          Show
          Adrian Cole (Inactive) added a comment - +1
          Hide
          Aled Sage added a comment -

          Should this issue be closed? Whirr 0.8.0 was released with a dependency on 1.5.0-beta.10.

          I've created WHIRR-666, asking for an upgrade to jclouds 1.5.1, ideally in Whirr 0.8.1.
          Or is the intent to keep this sole issue open until there is a whirr stable release that depends on a final jclouds 1.5.x release?

          Show
          Aled Sage added a comment - Should this issue be closed? Whirr 0.8.0 was released with a dependency on 1.5.0-beta.10. I've created WHIRR-666 , asking for an upgrade to jclouds 1.5.1, ideally in Whirr 0.8.1. Or is the intent to keep this sole issue open until there is a whirr stable release that depends on a final jclouds 1.5.x release?
          Hide
          Tom White added a comment -

          Closing since Whirr was updated to jclouds 1.5.0 in WHIRR-659.

          Show
          Tom White added a comment - Closing since Whirr was updated to jclouds 1.5.0 in WHIRR-659 .

            People

            • Assignee:
              Adrian Cole (Inactive)
              Reporter:
              Andrei Savu
            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development