Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      At this time 1.4.0 RC1 is out and if no major bugs are discovered we should be able to have a final release pretty soon.

      1. WHIRR-504.patch
        179 kB
        Andrei Savu
      2. WHIRR-504.patch
        175 kB
        Andrei Savu
      3. WHIRR-504.patch
        177 kB
        David Alves
      4. WHIRR-504.patch
        171 kB
        Adrian Cole
      5. WHIRR-504-minimal.patch
        0.7 kB
        Ioannis Canellos
      6. WHIRR-504.patch
        171 kB
        Adrian Cole
      7. WHIRR-504.patch
        178 kB
        Andrei Savu
      8. WHIRR-504.patch
        175 kB
        Adrian Cole
      9. WHIRR-504.patch
        3 kB
        Andrei Savu
      10. WHIRR-504.patch
        2 kB
        Andrei Savu

        Activity

        Hide
        Andrei Savu added a comment -

        Patch to upgrade to jclouds 1.4.0-rc.1. Seems to be working fine on cloudservers but fails to authenticate against aws-ec2 VMs.

        Show
        Andrei Savu added a comment - Patch to upgrade to jclouds 1.4.0-rc.1. Seems to be working fine on cloudservers but fails to authenticate against aws-ec2 VMs.
        Hide
        Andrei Savu added a comment -

        Update patch to build with 1.4.0-SNAPSHOT. Integration tests work fine on aws-ec2 & cloudservers-uk.

        Show
        Andrei Savu added a comment - Update patch to build with 1.4.0-SNAPSHOT. Integration tests work fine on aws-ec2 & cloudservers-uk.
        Hide
        Adrian Cole added a comment - - edited

        I looked over whirr for deprecation issues, as it uses things in guava that have replacements. Also, I had a look at DryRunModule, which is very sensitive to internal details of jclouds. I'm in the process of revamping this patch to get rid of all deprecated guava usage. I'm also updating jclouds to publish statement execution events on an EventBus, which is much cleaner than the aop pattern currently used in DryRunModule.

        Show
        Adrian Cole added a comment - - edited I looked over whirr for deprecation issues, as it uses things in guava that have replacements. Also, I had a look at DryRunModule, which is very sensitive to internal details of jclouds. I'm in the process of revamping this patch to get rid of all deprecated guava usage. I'm also updating jclouds to publish statement execution events on an EventBus, which is much cleaner than the aop pattern currently used in DryRunModule.
        Hide
        Adrian Cole added a comment -

        This patch updates to latest jclouds and also addresses all deprecated usage and all compiler warnings I could get rid of (save apache commons config, which aren't resolvable unless they update to generic types)

        Show
        Adrian Cole added a comment - This patch updates to latest jclouds and also addresses all deprecated usage and all compiler warnings I could get rid of (save apache commons config, which aren't resolvable unless they update to generic types)
        Hide
        Adrian Cole added a comment - - edited

        Updates in latest patch:

        • employs new jclouds features such as submitScriptOnNode and its EventBus mechanisms for script notification.
        • refactored DryRunModule and showed an example test you can use to sanity check services offline, based on David's work: CassandraServiceDryRunTest
        • refactored StatementBuilder to use new InitScript in jclouds
        • shown an example of how to use jclouds InstallJDK function in ZooKeeperClusterActionHandler.

        Results:

        • zookeeper passed on cloudservers-us, but when the zookeepers start on aws-ec2, the tests that actually connect to the ports fail.
        • I tried cassandra, but there seems to be an issue, even on rackspace where the default install_jdk script fails with "Package sun-java6-jdk has no installation candidate" I'd suggest to move to InstallJDK statement like I did in zookeeper.
        • The itests seem to rely on guava 10.0.1, which is a problem, maybe Ioannis can have a look and see if this can be updated.

        Note:
        I know I've reorganized imports and I understand that this isn't nice, but I don't have any more time, so I figured I'd kick the patch out for folks to help progress.

        Show
        Adrian Cole added a comment - - edited Updates in latest patch: employs new jclouds features such as submitScriptOnNode and its EventBus mechanisms for script notification. refactored DryRunModule and showed an example test you can use to sanity check services offline, based on David's work: CassandraServiceDryRunTest refactored StatementBuilder to use new InitScript in jclouds shown an example of how to use jclouds InstallJDK function in ZooKeeperClusterActionHandler. Results: zookeeper passed on cloudservers-us, but when the zookeepers start on aws-ec2, the tests that actually connect to the ports fail. I tried cassandra, but there seems to be an issue, even on rackspace where the default install_jdk script fails with "Package sun-java6-jdk has no installation candidate" I'd suggest to move to InstallJDK statement like I did in zookeeper. The itests seem to rely on guava 10.0.1, which is a problem, maybe Ioannis can have a look and see if this can be updated. Note: I know I've reorganized imports and I understand that this isn't nice, but I don't have any more time, so I figured I'd kick the patch out for folks to help progress.
        Hide
        Adrian Cole added a comment -

        slight update w/cleaner syntax to get the EventBus:

        context.utils().eventBus().register(ComputeCache.this);

        Show
        Adrian Cole added a comment - slight update w/cleaner syntax to get the EventBus: context.utils().eventBus().register(ComputeCache.this);
        Hide
        Andrei Savu added a comment -

        Just took a quick look. Nice work! I will do more review tomorrow. I see jclouds 1.4.0-rc.2 is now on Maven Central. What do you think about updating the Whirr trunk to this version? We should be able to release 1.4.0 pretty soon.

        Show
        Andrei Savu added a comment - Just took a quick look. Nice work! I will do more review tomorrow. I see jclouds 1.4.0-rc.2 is now on Maven Central. What do you think about updating the Whirr trunk to this version? We should be able to release 1.4.0 pretty soon.
        Hide
        Adrian Cole added a comment -

        +1 there aren't any api-related changes I'd expect between 1.4.0-rc.2 and final, and besides whirr feedback, I don't expect any bug related changes either.

        Show
        Adrian Cole added a comment - +1 there aren't any api-related changes I'd expect between 1.4.0-rc.2 and final, and besides whirr feedback, I don't expect any bug related changes either.
        Hide
        Andrei Savu added a comment -

        Karaf itests are failing for me and that's because PAX-EXAM needs guava 10.0.1. We may need to revert some changes to make things work. Ioannis any ideas? Is there a newer version of PAX-EXAM that works with Guava 11.0.1?

        Show
        Andrei Savu added a comment - Karaf itests are failing for me and that's because PAX-EXAM needs guava 10.0.1. We may need to revert some changes to make things work. Ioannis any ideas? Is there a newer version of PAX-EXAM that works with Guava 11.0.1?
        Hide
        Andrei Savu added a comment -

        It seem like pax-exam 2.4.0-RC1 is also using guava r09. Ioannis can we remove the whirr-core dependency from karaf itests?

        Show
        Andrei Savu added a comment - It seem like pax-exam 2.4.0-RC1 is also using guava r09. Ioannis can we remove the whirr-core dependency from karaf itests?
        Hide
        Andrei Savu added a comment -

        Ioannis can you explain the workaround you found for this one?

        Show
        Andrei Savu added a comment - Ioannis can you explain the workaround you found for this one?
        Hide
        Ioannis Canellos added a comment -

        Sorry for the delay.

        Ok, the problem is that itests require guava 11.x for compile and guava 10.x for test scope.
        The workaround I found is to remove the compile-time dependency from guava and let the test use guava 10.x.

        A secondary issue, was that the patch was using a snapshot version of jclouds and the current whirr setup did not support it.

        For the last I have created a new jira: WHIRR-536. It also fixes the guava thing, so if you apply it I think that all issues will be solved. Note: that when upgrading jclouds version, the osgi itests might fail if you don't also upgrade the jclouds.karaf.version.

        Show
        Ioannis Canellos added a comment - Sorry for the delay. Ok, the problem is that itests require guava 11.x for compile and guava 10.x for test scope. The workaround I found is to remove the compile-time dependency from guava and let the test use guava 10.x. A secondary issue, was that the patch was using a snapshot version of jclouds and the current whirr setup did not support it. For the last I have created a new jira: WHIRR-536 . It also fixes the guava thing, so if you apply it I think that all issues will be solved. Note: that when upgrading jclouds version, the osgi itests might fail if you don't also upgrade the jclouds.karaf.version.
        Hide
        Andrei Savu added a comment -

        Now that WHIRR-536 is in and 1.4.0-rc.3 available from maven central we need to update this patch. I will look into this later today.

        Show
        Andrei Savu added a comment - Now that WHIRR-536 is in and 1.4.0-rc.3 available from maven central we need to update this patch. I will look into this later today.
        Hide
        Andrei Savu added a comment -

        Updated patch for the current trunk. Unfortunately the OSGi tests fail and I was unable to find a fix in a reasonable amount of time. I will try again later.

        Show
        Andrei Savu added a comment - Updated patch for the current trunk. Unfortunately the OSGi tests fail and I was unable to find a fix in a reasonable amount of time. I will try again later.
        Hide
        Adrian Cole added a comment -

        @ioannis I updated the patch, as we are no longer in a staging repo, yet still fails starting with the following:
        org.ops4j.pax.exam.TestContainerException: [testInstallation(org.apache.whirr.karaf.itest.WhirrInstallationTest): Bundle org.apache.whirr.karaf.commands does not exist]
        can you have another look?

        Show
        Adrian Cole added a comment - @ioannis I updated the patch, as we are no longer in a staging repo, yet still fails starting with the following: org.ops4j.pax.exam.TestContainerException: [testInstallation(org.apache.whirr.karaf.itest.WhirrInstallationTest): Bundle org.apache.whirr.karaf.commands does not exist] can you have another look?
        Hide
        Ioannis Canellos added a comment -

        I am attaching an updated version of 504 patch, which fixes the OSGi related issues.

        Some comments about the nature of the problem. The maven bundle plugin normalizes the version when creating the bundles. The normalization it performs is something like splitting the version into major, minor, macro and classifier parts. It replaces character like '-' with '.' etc. Finally if the classifier part contains '.'s then it replaces them with _.

        I have added some ant regexp magic in order to cleanup the jclouds version and have it match with what bundle plugin does, so that we can use them in the fragment definitions (this parts are not covered by the bundle plugin). However, the regex I added can't cover the case were the classifier part contains a '.' e.g. rc.3.

        As a work around I am explicitly specifying the jclouds version 1.4.0.rc_3 as it should be.

        We could do 2 things in the future. Add some more regex goodness to properly convert the version or use a convention in the jclouds versioning that would keep us out of trouble: e.g. 1.4.0-rc3 (with no dots).

        Since the problem occurs just for the rc version of jclouds I think that explicitly specifying it is a viable solution for now.

        Show
        Ioannis Canellos added a comment - I am attaching an updated version of 504 patch, which fixes the OSGi related issues. Some comments about the nature of the problem. The maven bundle plugin normalizes the version when creating the bundles. The normalization it performs is something like splitting the version into major, minor, macro and classifier parts. It replaces character like '-' with '.' etc. Finally if the classifier part contains '.'s then it replaces them with _. I have added some ant regexp magic in order to cleanup the jclouds version and have it match with what bundle plugin does, so that we can use them in the fragment definitions (this parts are not covered by the bundle plugin). However, the regex I added can't cover the case were the classifier part contains a '.' e.g. rc.3. As a work around I am explicitly specifying the jclouds version 1.4.0.rc_3 as it should be. We could do 2 things in the future. Add some more regex goodness to properly convert the version or use a convention in the jclouds versioning that would keep us out of trouble: e.g. 1.4.0-rc3 (with no dots). Since the problem occurs just for the rc version of jclouds I think that explicitly specifying it is a viable solution for now.
        Hide
        Ioannis Canellos added a comment -

        I am attaching a supplemental patch that is to be applied on top of the last patch.

        Show
        Ioannis Canellos added a comment - I am attaching a supplemental patch that is to be applied on top of the last patch.
        Hide
        Andrei Savu added a comment -

        Thanks Ioannis! Let's wait for the final 1.4.0 release before committing this change.

        Show
        Andrei Savu added a comment - Thanks Ioannis! Let's wait for the final 1.4.0 release before committing this change.
        Hide
        Adrian Cole added a comment -

        On clean checkout of whirr, I get the following error. That said, maybe this patch will also correct this:

        Tests in error:
        testInstallation:WhirrInstallationTest.testInstallation:KarafTestContainer

        {mvn:org.apache.karaf\/apache-karaf\/2.2.5\/tar.gz}

        (org.apache.whirr.karaf.itest.WhirrInstallationTest): org/apache/whirr/karaf/itest/WhirrKarafTestSupport (wrong name: src/test/java/org/apache/whirr/karaf/itest/WhirrKarafTestSupport)
        testServices:WhirrServicesTest.testServices:KarafTestContainer

        {mvn:org.apache.karaf\/apache-karaf\/2.2.5\/tar.gz}

        (org.apache.whirr.karaf.itest.WhirrServicesTest): org/apache/whirr/karaf/itest/WhirrKarafTestSupport (wrong name: src/test/java/org/apache/whirr/karaf/itest/WhirrKarafTestSupport)

        Show
        Adrian Cole added a comment - On clean checkout of whirr, I get the following error. That said, maybe this patch will also correct this: Tests in error: testInstallation:WhirrInstallationTest.testInstallation:KarafTestContainer {mvn:org.apache.karaf\/apache-karaf\/2.2.5\/tar.gz} (org.apache.whirr.karaf.itest.WhirrInstallationTest): org/apache/whirr/karaf/itest/WhirrKarafTestSupport (wrong name: src/test/java/org/apache/whirr/karaf/itest/WhirrKarafTestSupport) testServices:WhirrServicesTest.testServices:KarafTestContainer {mvn:org.apache.karaf\/apache-karaf\/2.2.5\/tar.gz} (org.apache.whirr.karaf.itest.WhirrServicesTest): org/apache/whirr/karaf/itest/WhirrKarafTestSupport (wrong name: src/test/java/org/apache/whirr/karaf/itest/WhirrKarafTestSupport)
        Hide
        Adrian Cole added a comment -

        will test 1.4.0 from the following staging repositories:

        jclouds: https://oss.sonatype.org/content/repositories/orgjclouds-394/
        jclouds-karaf: https://oss.sonatype.org/content/repositories/orgjclouds-405/

        Show
        Adrian Cole added a comment - will test 1.4.0 from the following staging repositories: jclouds: https://oss.sonatype.org/content/repositories/orgjclouds-394/ jclouds-karaf: https://oss.sonatype.org/content/repositories/orgjclouds-405/
        Hide
        Adrian Cole added a comment -

        tested on aws-ec2 and cloudservers-uk on cassandra and zookeeper

        Show
        Adrian Cole added a comment - tested on aws-ec2 and cloudservers-uk on cassandra and zookeeper
        Hide
        Adrian Cole added a comment -

        p.s. the two errors I noted above on itests are are still present regardless of this patch

        Show
        Adrian Cole added a comment - p.s. the two errors I noted above on itests are are still present regardless of this patch
        Hide
        David Alves added a comment -

        Updated patch that applies cleanly to the latest trunk.
        two Karaf itests are failing:
        testInstallation:WhirrInstallationTest.testInstallation:KarafTestContainer

        {mvn:org.apache.karaf\/apache-karaf\/2.2.5\/tar.gz}

        (org.apache.whirr.karaf.itest.WhirrInstallationTest): Expected bundle to be started expected:<32> but was:<2>
        testServices:WhirrServicesTest.testServices:KarafTestContainer

        {mvn:org.apache.karaf\/apache-karaf\/2.2.5\/tar.gz}

        (org.apache.whirr.karaf.itest.WhirrServicesTest): Gave up waiting for service (&(objectClass=org.apache.whirr.ClusterController)(name=default))

        Show
        David Alves added a comment - Updated patch that applies cleanly to the latest trunk. two Karaf itests are failing: testInstallation:WhirrInstallationTest.testInstallation:KarafTestContainer {mvn:org.apache.karaf\/apache-karaf\/2.2.5\/tar.gz} (org.apache.whirr.karaf.itest.WhirrInstallationTest): Expected bundle to be started expected:<32> but was:<2> testServices:WhirrServicesTest.testServices:KarafTestContainer {mvn:org.apache.karaf\/apache-karaf\/2.2.5\/tar.gz} (org.apache.whirr.karaf.itest.WhirrServicesTest): Gave up waiting for service (&(objectClass=org.apache.whirr.ClusterController)(name=default))
        Hide
        Andrei Savu added a comment -

        Attached same patch with not functional changes only conflict fixes for latest trunk. I have also increased the SERVICE_TIMEOUT for the Karaf itests. Everything builds fine for me.

        Show
        Andrei Savu added a comment - Attached same patch with not functional changes only conflict fixes for latest trunk. I have also increased the SERVICE_TIMEOUT for the Karaf itests. Everything builds fine for me.
        Hide
        Andrei Savu added a comment -

        Note: before commit we need to update BootstrapTemplate to use addUserAndAuthorizeSudo (renamed to ensureUserExistsAndAthorizeSudo).

        Show
        Andrei Savu added a comment - Note: before commit we need to update BootstrapTemplate to use addUserAndAuthorizeSudo (renamed to ensureUserExistsAndAthorizeSudo).
        Hide
        Andrei Savu added a comment -

        Here is an updated version for current trunk. Seems to be working fine on aws-ec2 and cloudservers-uk. Please review!

        Changes:

        • updated BootstrapTemplate
        • removed whirr.hardware-min-ram=512 - I had some issue with t1.micro
        Show
        Andrei Savu added a comment - Here is an updated version for current trunk. Seems to be working fine on aws-ec2 and cloudservers-uk. Please review! Changes: updated BootstrapTemplate removed whirr.hardware-min-ram=512 - I had some issue with t1.micro
        Hide
        Andrei Savu added a comment -

        David, Karel do you have some time to review? This is quite a big and disruptive change.

        Show
        Andrei Savu added a comment - David, Karel do you have some time to review? This is quite a big and disruptive change.
        Hide
        Karel Vervaeke added a comment -

        Basic unit tests work (mvn install). Integration test for hadoop on ec2 works.
        I'll have a closer look at the changes later.

        Show
        Karel Vervaeke added a comment - Basic unit tests work (mvn install). Integration test for hadoop on ec2 works. I'll have a closer look at the changes later.
        Hide
        David Alves added a comment -

        +1

        • applies cleanly to trunk
        • unit tests pass
        • i'm cloudless for the day so itested zookeeper with vbox (whirr-379) and runs fine.
        Show
        David Alves added a comment - +1 applies cleanly to trunk unit tests pass i'm cloudless for the day so itested zookeeper with vbox (whirr-379) and runs fine.
        Hide
        Karel Vervaeke added a comment -

        +1 looks great

        Show
        Karel Vervaeke added a comment - +1 looks great
        Hide
        Andrei Savu added a comment -

        Committed to trunk. Thanks guys for reviewing and for all the help on the way! Thanks Adrian!

        Show
        Andrei Savu added a comment - Committed to trunk. Thanks guys for reviewing and for all the help on the way! Thanks Adrian!

          People

          • Assignee:
            Adrian Cole
            Reporter:
            Andrei Savu
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development