Whirr
  1. Whirr
  2. WHIRR-6

Write a Rackspace cloud provider

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: contrib/python
    • Labels:
      None

      Description

      The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.

      1. HADOOP-6464.patch
        47 kB
        Tom White
      2. HADOOP-6464.patch
        44 kB
        Tom White
      3. HADOOP-6464.patch
        41 kB
        Tom White

        Activity

        Hide
        Tom White added a comment -

        This patch uses the Python libcloud API (http://incubator.apache.org/libcloud/) to communicate with Rackspace's API. Instrcutions are provided in the README.

        Show
        Tom White added a comment - This patch uses the Python libcloud API ( http://incubator.apache.org/libcloud/ ) to communicate with Rackspace's API. Instrcutions are provided in the README.
        Hide
        Aaron Kimball added a comment -

        Can you please describe the test plan you used to actually test end-to-end coverage of rackspace?

        regarding the patch:

        see comments re. TERREMARK_KEY in HADOOP-6465 for RACKSPACE_KEY, etc.

        re. boot script: I noticed a lot of very similar code in the terremark boot script. Can some of this be factored out into a common utility script that gets sourced in? This seems to be a looming maintenence problem.

        RackspaceCluster.wait_for_instances() looks like a clone of TerremarkCluster.wait_for_instances(). Can this be pushed up into cluster?

        Ditto with _all_started(), _ip_list_to_string(), terminate()?

        Why is rackspace remote-setup.sh strictly-ubuntu-based? The boot script looks like it also handles centos.

        Show
        Aaron Kimball added a comment - Can you please describe the test plan you used to actually test end-to-end coverage of rackspace? regarding the patch: see comments re. TERREMARK_KEY in HADOOP-6465 for RACKSPACE_KEY, etc. re. boot script: I noticed a lot of very similar code in the terremark boot script. Can some of this be factored out into a common utility script that gets sourced in? This seems to be a looming maintenence problem. RackspaceCluster.wait_for_instances() looks like a clone of TerremarkCluster.wait_for_instances(). Can this be pushed up into cluster? Ditto with _all_started(), _ip_list_to_string(), terminate()? Why is rackspace remote-setup.sh strictly-ubuntu-based? The boot script looks like it also handles centos.
        Hide
        Tom White added a comment -

        > Can you please describe the test plan you used to actually test end-to-end coverage of rackspace?

        I tested this by manually spinning up a cluster, running a job, checking the results and terminating the cluster. The commands to do this are in the README.

        I've now modified the integration test script to run on Rackspace, which I successfully ran with the following (note that the image ID is private, since Rackspace does not support shared images):

        HADOOP_CLOUD_PROVIDER=rackspace IMAGE_ID=200181 INSTANCE_TYPE=4 HADOOP_HOME=~/dev/hadoop-0.20.1/ ./transient-cluster.sh
        

        > regarding the patch:
        > see comments re. TERREMARK_KEY in HADOOP-6465 for RACKSPACE_KEY, etc.

        Agree that this would be a useful way to specify keys. I've created HADOOP-6480 for this.

        > re. boot script: I noticed a lot of very similar code in the terremark boot script. Can some of this be factored out into a common utility script that gets sourced in? This seems to be a looming maintenence problem.

        This could become a problem, but I'd like to address it in a separate issue.

        > RackspaceCluster.wait_for_instances() looks like a clone of TerremarkCluster.wait_for_instances(). Can this be pushed up into cluster?
        > Ditto with _all_started(), _ip_list_to_string(), terminate()?

        These are the same methods. I'd like to commit this one first, then I'll re-work HADOOP-6465 so the methods are pulled up into Cluster.

        > Why is rackspace remote-setup.sh strictly-ubuntu-based? The boot script looks like it also handles centos.

        I've only tested on Ubuntu so far, but it would be straightforward to run on Centos too. I'll create another JIRA for this.

        Thanks for the review!

        Show
        Tom White added a comment - > Can you please describe the test plan you used to actually test end-to-end coverage of rackspace? I tested this by manually spinning up a cluster, running a job, checking the results and terminating the cluster. The commands to do this are in the README. I've now modified the integration test script to run on Rackspace, which I successfully ran with the following (note that the image ID is private, since Rackspace does not support shared images): HADOOP_CLOUD_PROVIDER=rackspace IMAGE_ID=200181 INSTANCE_TYPE=4 HADOOP_HOME=~/dev/hadoop-0.20.1/ ./ transient -cluster.sh > regarding the patch: > see comments re. TERREMARK_KEY in HADOOP-6465 for RACKSPACE_KEY, etc. Agree that this would be a useful way to specify keys. I've created HADOOP-6480 for this. > re. boot script: I noticed a lot of very similar code in the terremark boot script. Can some of this be factored out into a common utility script that gets sourced in? This seems to be a looming maintenence problem. This could become a problem, but I'd like to address it in a separate issue. > RackspaceCluster.wait_for_instances() looks like a clone of TerremarkCluster.wait_for_instances(). Can this be pushed up into cluster? > Ditto with _all_started(), _ip_list_to_string(), terminate()? These are the same methods. I'd like to commit this one first, then I'll re-work HADOOP-6465 so the methods are pulled up into Cluster. > Why is rackspace remote-setup.sh strictly-ubuntu-based? The boot script looks like it also handles centos. I've only tested on Ubuntu so far, but it would be straightforward to run on Centos too. I'll create another JIRA for this. Thanks for the review!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12429556/HADOOP-6464.patch
        against trunk revision 896259.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        -1 release audit. The applied patch generated 4 release audit warnings (more than the trunk's current 1 warnings).

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/testReport/
        Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429556/HADOOP-6464.patch against trunk revision 896259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 4 release audit warnings (more than the trunk's current 1 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/console This message is automatically generated.
        Hide
        Tom White added a comment -

        This version of the patch should fix the release audit warnings. I've moved the Rackspace documentation into the top-level README (at some point the documentation should be broken out into a more structured format).

        Show
        Tom White added a comment - This version of the patch should fix the release audit warnings. I've moved the Rackspace documentation into the top-level README (at some point the documentation should be broken out into a more structured format).
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12429565/HADOOP-6464.patch
        against trunk revision 896259.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429565/HADOOP-6464.patch against trunk revision 896259. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/console This message is automatically generated.
        Hide
        Aaron Kimball added a comment -

        Your intended refactorings sound reasonable. +1 on the patch

        Show
        Aaron Kimball added a comment - Your intended refactorings sound reasonable. +1 on the patch
        Hide
        Tom White added a comment -

        I've just committed this.

        Show
        Tom White added a comment - I've just committed this.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #138 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/138/)
        . Write a Rackspace cloud provider.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #138 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/138/ ) . Write a Rackspace cloud provider.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #212 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/212/)
        . Write a Rackspace cloud provider.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #212 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/212/ ) . Write a Rackspace cloud provider.

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development