Whirr
  1. Whirr
  2. WHIRR-681

enhance puppet service with an ability to export cluster topology to the puppet code

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.1
    • Fix Version/s: 0.8.2, 0.9.0
    • Component/s: service/puppet
    • Labels:
      None

      Description

      I would like to propose that we enhance puppet service (btw, there seems to be no service/puppet in the components list) by implement the capability that exports the cluster topology via extlookup queryable manifest that gets added to all the nodes same way site.pp does.

      1. WHIRR-681.patch.txt
        6 kB
        Roman Shaposhnik
      2. WHIRR-681-2.patch.txt
        9 kB
        Roman Shaposhnik
      3. WHIRR-681-3.patch.txt
        9 kB
        Roman Shaposhnik

        Issue Links

          Activity

          Hide
          Andrew Bayer added a comment -

          Pushed to branch-0.8.

          Show
          Andrew Bayer added a comment - Pushed to branch-0.8.
          Hide
          Andrew Bayer added a comment -

          Committed.

          Show
          Andrew Bayer added a comment - Committed.
          Hide
          Andrew Bayer added a comment -

          +1 - looks good to me. If I don't hear anything by next week, I'm gonna commit this.

          Show
          Andrew Bayer added a comment - +1 - looks good to me. If I don't hear anything by next week, I'm gonna commit this.
          Hide
          Tom White added a comment -

          Services in Whirr don't constitute a part of of Whirr's public API, so I think it's fine to change the constructor.

          Show
          Tom White added a comment - Services in Whirr don't constitute a part of of Whirr's public API, so I think it's fine to change the constructor.
          Hide
          Roman Shaposhnik added a comment -

          Updating the patch so that the location of the extlookup data gets deposited into the generated site.pp. Initially I wanted to make it tweakable from within the modules, but it seems that Puppet makes it really difficult to do so.

          Show
          Roman Shaposhnik added a comment - Updating the patch so that the location of the extlookup data gets deposited into the generated site.pp. Initially I wanted to make it tweakable from within the modules, but it seems that Puppet makes it really difficult to do so.
          Hide
          Roman Shaposhnik added a comment -

          Good question – personally I'd say there's very little chance of other projects depending on this particular one, but I'd love to hear what rules apply to statements of individual Whirr services.

          Show
          Roman Shaposhnik added a comment - Good question – personally I'd say there's very little chance of other projects depending on this particular one, but I'd love to hear what rules apply to statements of individual Whirr services.
          Hide
          Mark Grover added a comment -

          A minor question here:
          The interface being updated in this patch CreateSitePpAndApplyRoles is a public one. While I do see all references in Whirr code itself have been updated to match up with the new signature of the constructor, my question is should we create a new constructor instead of modifying the existing one to maintain compatibility with any existing 3rd party code outside of Whirr codebase that might be using it?

          Show
          Mark Grover added a comment - A minor question here: The interface being updated in this patch CreateSitePpAndApplyRoles is a public one. While I do see all references in Whirr code itself have been updated to match up with the new signature of the constructor, my question is should we create a new constructor instead of modifying the existing one to maintain compatibility with any existing 3rd party code outside of Whirr codebase that might be using it?
          Hide
          Tom White added a comment -

          Thanks for the explanation. Is there a Bigtop JIRA to cross reference? Once that is done having a bit of documentation on how to use the two together would be very useful.

          The unit test looks fine to me.

          I've assigned this JIRA to you now.

          Show
          Tom White added a comment - Thanks for the explanation. Is there a Bigtop JIRA to cross reference? Once that is done having a bit of documentation on how to use the two together would be very useful. The unit test looks fine to me. I've assigned this JIRA to you now.
          Hide
          Roman Shaposhnik added a comment - - edited

          Re: CreateSitePpAndApplyRolesTest – absolutely! Attached is a new patch with the unit test updated. Sorry for not updating it to begin with.

          As to how this feature is going to be used (and perhaps we need to put this in docs some place) – it allows individual puppet agents discover the topology of the cluster (IOW, what classes are applied to what nodes) and makes decisions based on that. Here's how it will work on the puppet side (using Zookeeper as an example):

          $ensemble = extlookup('zookeeper::server', ['127.0.0.1'])
          ...
          and later on in the configuration file template:
          <% ensemble.each_with_index do |server,idx| %>
          server.<%= idx %>=<%= server %>
          <% end %>
          

          That's the general idea.

          Now, I'm still tweaking this patch as far as the best integration with Puppet is concerned (hence not putting it into a 'Patch Submitted' state but simply asking for early feedback). I'm rewriting Bigtop Puppet code as part of this as well.

          Finally, as for WHIRR-516 – this JIRA is required to make progress on WHIRR-516. My ultimate goal as far as WHIRR-516 is concerned is to be able to deploy Bigtop clusters by a simple:

          whirr.instance-templates=1 puppet:bigtop-hadoop-hdfs::namenode+\
                                     puppet:bigtop-hbase::master+\
                                   4 puppet:bigtop-zookeeper::server+\
                                     puppet:bigtop-hadoop-hdfs::datanode+\
                                     puppet:bigtop-hbase::server
          

          Hope this answers your questions and, once again, thanks for the feedback – much appreciated!

          P.S. Feel free to add me to the project so I can assign these JIRAs to myself.

          Show
          Roman Shaposhnik added a comment - - edited Re: CreateSitePpAndApplyRolesTest – absolutely! Attached is a new patch with the unit test updated. Sorry for not updating it to begin with. As to how this feature is going to be used (and perhaps we need to put this in docs some place) – it allows individual puppet agents discover the topology of the cluster (IOW, what classes are applied to what nodes) and makes decisions based on that. Here's how it will work on the puppet side (using Zookeeper as an example): $ensemble = extlookup('zookeeper::server', ['127.0.0.1']) ... and later on in the configuration file template: <% ensemble.each_with_index do |server,idx| %> server.<%= idx %>=<%= server %> <% end %> That's the general idea. Now, I'm still tweaking this patch as far as the best integration with Puppet is concerned (hence not putting it into a 'Patch Submitted' state but simply asking for early feedback). I'm rewriting Bigtop Puppet code as part of this as well. Finally, as for WHIRR-516 – this JIRA is required to make progress on WHIRR-516 . My ultimate goal as far as WHIRR-516 is concerned is to be able to deploy Bigtop clusters by a simple: whirr.instance-templates=1 puppet:bigtop-hadoop-hdfs::namenode+\ puppet:bigtop-hbase::master+\ 4 puppet:bigtop-zookeeper::server+\ puppet:bigtop-hadoop-hdfs::datanode+\ puppet:bigtop-hbase::server Hope this answers your questions and, once again, thanks for the feedback – much appreciated! P.S. Feel free to add me to the project so I can assign these JIRAs to myself.
          Hide
          Tom White added a comment -

          Thanks Roman, this looks great! Could you elaborate on how you use this feature? Also, is it basically equivalent to WHIRR-516?

          It would be good to add a unit test for the new logic in CreateSitePpAndApplyRolesTest.

          Show
          Tom White added a comment - Thanks Roman, this looks great! Could you elaborate on how you use this feature? Also, is it basically equivalent to WHIRR-516 ? It would be good to add a unit test for the new logic in CreateSitePpAndApplyRolesTest.
          Hide
          Roman Shaposhnik added a comment -

          Here's a pretty trivial patch. Please review.

          Show
          Roman Shaposhnik added a comment - Here's a pretty trivial patch. Please review.
          Hide
          Guido Serra aka Zeph added a comment -

          Hi, I'm not sure if this is a clone of WHIRR-255

          Show
          Guido Serra aka Zeph added a comment - Hi, I'm not sure if this is a clone of WHIRR-255

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development