Whirr
  1. Whirr
  2. WHIRR-332

Need to specify different instance size/type depending on role

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.8.0
    • Component/s: service/hadoop
    • Labels:
      None

      Description

      When setting up Hadoop, whirr.hardware-id is used in the configuration to set a single instance size/type for both nn,jt and dn,tt roles, but there are workloads that benefit from having a different instance size/type for each role. This need is even greater for ensembles with Zookeeper and HBase.

      Without this, a small Whirr cluster underutilizes resources.

      whirr.hardware-id should be a used to set a default instance size/type, and variations like whirr.$

      {role}

      .hardware-id could be used to set size/type more specifically. This is a simple, obvious, and easy to implement mechanism to provide this functionality.

      1. WHIRR-332.patch
        36 kB
        Andrei Savu
      2. WHIRR-332.patch
        36 kB
        Andrei Savu
      3. WHIRR-332.patch
        26 kB
        Andrei Savu
      4. WHIRR-332.patch
        21 kB
        Andrei Savu

        Issue Links

          Activity

          Hide
          Andrei Savu added a comment -

          Probably we will handle this in WHIRR-318 while creating a new format for specifying the cluster (a more flexible one).

          Show
          Andrei Savu added a comment - Probably we will handle this in WHIRR-318 while creating a new format for specifying the cluster (a more flexible one).
          Hide
          Tom White added a comment -

          This would be a useful improvement. I don't think it is dependent on WHIRR-318, but the property naming should be designed in the context of that JIRA.

          E.g. should it be whirr.templates.roles.$

          {role}

          .hardware-id, or perhaps even roles: whirr.templates.roles.hadoop-namenode+hadoop-jobtracker.hardware-id.

          Show
          Tom White added a comment - This would be a useful improvement. I don't think it is dependent on WHIRR-318 , but the property naming should be designed in the context of that JIRA. E.g. should it be whirr.templates.roles.$ {role} .hardware-id, or perhaps even roles : whirr.templates.roles.hadoop-namenode+hadoop-jobtracker.hardware-id.
          Hide
          Andrei Savu added a comment -

          I believe that we need something like: whirr.templates.roles.$

          {template-group}

          .hardware-id

          Show
          Andrei Savu added a comment - I believe that we need something like: whirr.templates.roles.$ {template-group} .hardware-id
          Hide
          Andrei Savu added a comment -

          Let's tackle this for 0.8.0. Maybe we can also look into adding the ability to use regular and spot instances at the same time (e.g. a Hadoop cluster with 50% regular 50% spot - I know EMR can do this trick).

          Show
          Andrei Savu added a comment - Let's tackle this for 0.8.0. Maybe we can also look into adding the ability to use regular and spot instances at the same time (e.g. a Hadoop cluster with 50% regular 50% spot - I know EMR can do this trick).
          Hide
          Andrei Savu added a comment -

          A first try on this one. Seems to be working fine on EC2.

          Show
          Andrei Savu added a comment - A first try on this one. Seems to be working fine on EC2.
          Hide
          Andrei Savu added a comment -

          In this patch:

          • removed deprecated code for role name aliases (e.g. nn -> hadoop-namenode)
          • cleanups + tests

          I'm thinking about adding support for specifying the spot price per instance templates in a new JIRA.

          Please review.

          Show
          Andrei Savu added a comment - In this patch: removed deprecated code for role name aliases (e.g. nn -> hadoop-namenode) cleanups + tests I'm thinking about adding support for specifying the spot price per instance templates in a new JIRA. Please review.
          Hide
          Tom White added a comment -

          I would remove the deprecated role name aliases in a separate JIRA so that folks scanning release notes see it, since it's an incompatible change.

          The test should check that roles that haven't been configured to use a particular type of hardware fall back to the default. Also, in testHardwareIdPerInstanceTemplate, what if only role1 has its hardware configured (i.e. not role1+role2) - it should fail or warn.

          Looks good otherwise.

          Show
          Tom White added a comment - I would remove the deprecated role name aliases in a separate JIRA so that folks scanning release notes see it, since it's an incompatible change. The test should check that roles that haven't been configured to use a particular type of hardware fall back to the default. Also, in testHardwareIdPerInstanceTemplate, what if only role1 has its hardware configured (i.e. not role1+role2) - it should fail or warn. Looks good otherwise.
          Hide
          Andrei Savu added a comment -

          Thanks Tom for reviewing. I have extracted a part in WHIRR-458 and I will address the remaining issues here.

          Show
          Andrei Savu added a comment - Thanks Tom for reviewing. I have extracted a part in WHIRR-458 and I will address the remaining issues here.
          Hide
          Andrei Savu added a comment -

          Added more tests and strict error handling. It should be good enough now.

          Show
          Andrei Savu added a comment - Added more tests and strict error handling. It should be good enough now.
          Hide
          Andrei Savu added a comment -

          Updated patch for latest trunk.

          Show
          Andrei Savu added a comment - Updated patch for latest trunk.
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Andrei Savu added a comment -

          Committed. Thanks Tom for reviewing.

          Show
          Andrei Savu added a comment - Committed. Thanks Tom for reviewing.

            People

            • Assignee:
              Andrei Savu
              Reporter:
              Paul Baclace
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development