Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: None
    • Labels:
      None

      Description

      note java support is via jclouds, scheduled for beta-10 release

      https://github.com/jclouds/jclouds-examples/tree/master/ec2-computeservice-spot

      1. WHIRR-260.patch
        8 kB
        Andrei Savu
      2. WHIRR-260.patch
        8 kB
        Andrei Savu
      3. WHIRR-260.patch
        9 kB
        Andrei Savu
      4. WHIRR-260.patch
        9 kB
        Andrei Savu

        Activity

        Hide
        Soren Macbeth added a comment -

        yes please!

        Show
        Soren Macbeth added a comment - yes please!
        Hide
        Andrei Savu added a comment -

        Attached a patch for this feature. I've run a simple manual test by running a Hadoop cluster. Let me know what you think.

        Show
        Andrei Savu added a comment - Attached a patch for this feature. I've run a simple manual test by running a Hadoop cluster. Let me know what you think.
        Hide
        Tom White added a comment -

        Should we put the cloud provider name into options that are only supported there? E.g. ec2_spot_price?

        It would be better to push the spot price configuration into TemplateBuilderStrategy, but that would mean changing that class to return Template, not TemplateBuilder. Given that we just call build() on it anyway, perhaps that's OK.

        What is the behaviour if the spot price can't be fulfilled? Does it time out? It would be nice to add a little documentation and/or links to any relevant EC2 docs.

        Show
        Tom White added a comment - Should we put the cloud provider name into options that are only supported there? E.g. ec2_spot_price? It would be better to push the spot price configuration into TemplateBuilderStrategy, but that would mean changing that class to return Template, not TemplateBuilder. Given that we just call build() on it anyway, perhaps that's OK. What is the behaviour if the spot price can't be fulfilled? Does it time out? It would be nice to add a little documentation and/or links to any relevant EC2 docs.
        Hide
        Tom White added a comment -

        I noticed that the instance size changed in the recipe - was that intentional?

        Show
        Tom White added a comment - I noticed that the instance size changed in the recipe - was that intentional?
        Hide
        Andrei Savu added a comment -

        Thanks Tom for reviewing. I will resubmit the patch tomorrow.

        I noticed that the instance size changed in the recipe - was that intentional?

        No. I will revert the change.

        Show
        Andrei Savu added a comment - Thanks Tom for reviewing. I will resubmit the patch tomorrow. I noticed that the instance size changed in the recipe - was that intentional? No. I will revert the change.
        Hide
        Andrei Savu added a comment -

        In this patch I've renamed whirr.spot-price to whirr.ec2-spot-price.

        Adrian is there a place where I can read more about how jclouds handles spot instances (except the source code)? I'm interested in learning how error conditions are handled.

        Show
        Andrei Savu added a comment - In this patch I've renamed whirr.spot-price to whirr.ec2-spot-price. Adrian is there a place where I can read more about how jclouds handles spot instances (except the source code)? I'm interested in learning how error conditions are handled.
        Hide
        Andrei Savu added a comment -

        I guess there is no answer to this question. I will start to dig through the source code.

        Show
        Andrei Savu added a comment - I guess there is no answer to this question. I will start to dig through the source code.
        Hide
        Adrian Cole added a comment -

        sorry. there is no guide outside the source code and amazon's documentation,
        http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/index.html?query-apis.html

        I generally recommend looking at the test cases for usage example, or posting to the jclouds-dev google group, as I'm better at following-up there, and also others can benefit from the questions. The more interesting test case to view is SpotInstanceClientLiveTest, which uses the api directly and AWSEC2ComputeServiceLiveTest.testExtendedOptionsAndLogin() which uses the spot price option on the portable interface.

        In the code, AWSEC2CreateNodesInGroupThenAddToSet actually manages creating the spot instance request. The way jclouds manages this is that it looks for both regular reservations and also spot requests when listing nodes. The convergence of these 2 is what you'll see in a listNodes command (AWSEC2ListNodesStrategy ex. does these multiple listings in parallel)

        If there's an error on the parameters requesting nodes, you'll receive an HttpResponseException in the 400 range from ec2 itself. The other condition you should be aware of is when the spot request goes through, but perhaps takes longer to provision the nodes than the jclouds default timeout (jclouds.compute.timeout.node-running), something I've not seen happen.

        I've added some of these notes here, and feel free to revise: http://code.google.com/p/jclouds/wiki/EC2

        Backtype were the first users of the Spot Price feature, last year. I can connect you, if you want some behavioral experience with this in the wild. Cheers!
        -A

        Show
        Adrian Cole added a comment - sorry. there is no guide outside the source code and amazon's documentation, http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/index.html?query-apis.html I generally recommend looking at the test cases for usage example, or posting to the jclouds-dev google group, as I'm better at following-up there, and also others can benefit from the questions. The more interesting test case to view is SpotInstanceClientLiveTest, which uses the api directly and AWSEC2ComputeServiceLiveTest.testExtendedOptionsAndLogin() which uses the spot price option on the portable interface. In the code, AWSEC2CreateNodesInGroupThenAddToSet actually manages creating the spot instance request. The way jclouds manages this is that it looks for both regular reservations and also spot requests when listing nodes. The convergence of these 2 is what you'll see in a listNodes command (AWSEC2ListNodesStrategy ex. does these multiple listings in parallel) If there's an error on the parameters requesting nodes, you'll receive an HttpResponseException in the 400 range from ec2 itself. The other condition you should be aware of is when the spot request goes through, but perhaps takes longer to provision the nodes than the jclouds default timeout (jclouds.compute.timeout.node-running), something I've not seen happen. I've added some of these notes here, and feel free to revise: http://code.google.com/p/jclouds/wiki/EC2 Backtype were the first users of the Spot Price feature, last year. I can connect you, if you want some behavioral experience with this in the wild. Cheers! -A
        Hide
        Andrei Savu added a comment -

        Thanks for the pointers. I would love an intro to Backtype because I know they've created a patch in the past for the Python contrib WHIRR-76. Maybe they are now interested in using / contributing to the Java version.

        Show
        Andrei Savu added a comment - Thanks for the pointers. I would love an intro to Backtype because I know they've created a patch in the past for the Python contrib WHIRR-76 . Maybe they are now interested in using / contributing to the Java version.
        Hide
        Tom White added a comment -

        So it sounds like if the price isn't fulfilled it times out after 20 minutes (jclouds.compute.timeout.node-running). This is the thing I was missing.

        Show
        Tom White added a comment - So it sounds like if the price isn't fulfilled it times out after 20 minutes (jclouds.compute.timeout.node-running). This is the thing I was missing.
        Hide
        Adrian Cole added a comment - - edited

        ah, yes. so in backtype, they always set spot price to current real price, so this doesn't happen [most of the time :)], and they always save money as you only pay the spot price, not the max value you put in.

        Show
        Adrian Cole added a comment - - edited ah, yes. so in backtype, they always set spot price to current real price, so this doesn't happen [most of the time :)] , and they always save money as you only pay the spot price, not the max value you put in.
        Hide
        Andrei Savu added a comment -

        I will update the patch to document the timeout behavior and it should be ready.

        Show
        Andrei Savu added a comment - I will update the patch to document the timeout behavior and it should be ready.
        Hide
        Soren Macbeth added a comment -

        Hi Guys,

        AWS was having issues last night filling spot requests, possibly related to the change in pricing by AZ rather then region (just speculation on my part), in an case, I tested the patch out while this was happening. Once the requests timed out, there was an issue where the previously unfilled spot requests weren't canceled, and entirely new set of requests were issued. This would be undesirable behaviour, so if possible whirr should catch the timeouts and clean up any outstanding open, but unfilled spot requests and cancel them.

        FYI - We (Yieldbot) also use spot based instances for our production hadoop clusters. We use a custom forked version of the python contrib scripts + a patch for spot instances based on BackType's work. We also use the trick of setting our bid price above the actual reserved instance price, but this isn't 100% successful, price spikes can still take out some instances, or prevent requests being filled, but it works well most of the time.

        Show
        Soren Macbeth added a comment - Hi Guys, AWS was having issues last night filling spot requests, possibly related to the change in pricing by AZ rather then region (just speculation on my part), in an case, I tested the patch out while this was happening. Once the requests timed out, there was an issue where the previously unfilled spot requests weren't canceled, and entirely new set of requests were issued. This would be undesirable behaviour, so if possible whirr should catch the timeouts and clean up any outstanding open, but unfilled spot requests and cancel them. FYI - We (Yieldbot) also use spot based instances for our production hadoop clusters. We use a custom forked version of the python contrib scripts + a patch for spot instances based on BackType's work. We also use the trick of setting our bid price above the actual reserved instance price, but this isn't 100% successful, price spikes can still take out some instances, or prevent requests being filled, but it works well most of the time.
        Hide
        Adrian Cole added a comment -

        soren .. great timing on that comment, thanks! we can update docs w/ recommendations based on your exp.

        Show
        Adrian Cole added a comment - soren .. great timing on that comment, thanks! we can update docs w/ recommendations based on your exp.
        Hide
        Andrei Savu added a comment -

        Updated patch documentation. It should be clear enough how this option works if you know what spot instances are.

        Show
        Andrei Savu added a comment - Updated patch documentation. It should be clear enough how this option works if you know what spot instances are.
        Hide
        Andrei Savu added a comment -

        What do you think? I'm planning to commit this tomorrow.

        Show
        Andrei Savu added a comment - What do you think? I'm planning to commit this tomorrow.
        Hide
        Andrei Savu added a comment -

        Changes suggested by Adrian Cole on chat:

        • it is more AWSEC2Client than EC2Client to check for because eucalyptus doesn't support a spot market and still uses EC2Client
        • update namespace from ec2 to aws-ec2
        Show
        Andrei Savu added a comment - Changes suggested by Adrian Cole on chat: it is more AWSEC2Client than EC2Client to check for because eucalyptus doesn't support a spot market and still uses EC2Client update namespace from ec2 to aws-ec2
        Hide
        Andrei Savu added a comment -

        Updated patch as discussed with Adrian. I'm going to commit and we can fix any remaining issues in new JIRAs.

        Show
        Andrei Savu added a comment - Updated patch as discussed with Adrian. I'm going to commit and we can fix any remaining issues in new JIRAs.
        Hide
        Andrei Savu added a comment -

        I've just committed this. Thanks guys for feedback.

        Show
        Andrei Savu added a comment - I've just committed this. Thanks guys for feedback.

          People

          • Assignee:
            Andrei Savu
            Reporter:
            Adrian Cole
          • Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development