Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-840

Refactor Bigtop puppet code for better integration with newer Puppet and Whirr

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Won't Fix
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.8.0
    • Component/s: deployment
    • Labels:
      None

      Description

      Our current puppet code is now showing its age (after all it was originally developed to run on, gasp, Puppet 0.25!). It would be nice to re-factor it so we can take care of the following shortcomings:

      1. namespace all of our code under bigtop and get rid of extra hadoop- prefixes. For example, zookeeper server class will be accessible as bigtop::zookeeper::server
      2. as a consequence of doing #1 we will be consolidating all of our code into a single bigtop module. Now, this could be controversial at first, but lets face – Bigtop is an integration platform so chance of somebody using our zookeeper module with Adobe's hadoop module to deploy a cluster are pretty low. Also, moving everything into a single module will make it way easier for us to register it at Puppet Forge (http://forge.puppetlabs.com/) and engage additional community that way
      3. make all of our code based on parameterized classes and get rid of all the dynamic scope lookups (which is deprecated in Puppet 3.0 anyway) and global variable settings.
      4. separate data from puppet code as descibed over here: https://puppetlabs.com/blog/the-problem-with-separating-data-from-puppet-code/
      5. pursue the roles/profiles class paradigm http://www.craigdunn.org/2012/05/239/

      If this still sounds pretty opaque to you, I'll post some POC code over the weekend so you can take a closer look.

      Also, if you have any other Puppet concerns – please jump right in – now is a good time to do a bit of spring cleaning.

      Finally, given that we will be introducing a brand new module (bigtop) all of our existing Puppet code will remain in place at least till Bigtop 0.7.0. By that time if the new one works well, we can deprecate the old one.

      1. BIGTOP-840-0.patch.txt
        14 kB
        Roman Shaposhnik

        Issue Links

          Activity

          Hide
          rvs Roman Shaposhnik added a comment -

          Attaching an example for how zookeeper puppet code is going to look like with the new re-factored puppet code.

          Please let me know what do you think.

          Show
          rvs Roman Shaposhnik added a comment - Attaching an example for how zookeeper puppet code is going to look like with the new re-factored puppet code. Please let me know what do you think.
          Hide
          bmahe Bruno Mahé added a comment -

          All of this sound like great improvements! But is all of this compatible with the puppet provided by our supported GNU/Linux distributions?
          From http://pkgs.org/search/?keyword=puppet the lowest common denominator remains puppet 2.6 on CentOS 5/6.

          Show
          bmahe Bruno Mahé added a comment - All of this sound like great improvements! But is all of this compatible with the puppet provided by our supported GNU/Linux distributions? From http://pkgs.org/search/?keyword=puppet the lowest common denominator remains puppet 2.6 on CentOS 5/6.
          Hide
          rvs Roman Shaposhnik added a comment -

          Well, the more I work with Puppet the more it feels to me that the most reliable way is, unfortunately, to use the puppet builds for each OS from Puppet Labs repos: http://yum.puppetlabs.com/ and http://apt.puppetlabs.com/

          Essentially I had to extend Whirr to make it happen since I could just rely on whatever happens to be the system puppet

          Show
          rvs Roman Shaposhnik added a comment - Well, the more I work with Puppet the more it feels to me that the most reliable way is, unfortunately, to use the puppet builds for each OS from Puppet Labs repos: http://yum.puppetlabs.com/ and http://apt.puppetlabs.com/ Essentially I had to extend Whirr to make it happen since I could just rely on whatever happens to be the system puppet
          Hide
          aklochkov Andrey Klochkov added a comment -

          While this refactoring is in progress, here's a simple patch which gets rid of dynamic scoping and makes bigtop-deploy working with Puppet 3.x: https://issues.apache.org/jira/browse/BIGTOP-1047

          Show
          aklochkov Andrey Klochkov added a comment - While this refactoring is in progress, here's a simple patch which gets rid of dynamic scoping and makes bigtop-deploy working with Puppet 3.x: https://issues.apache.org/jira/browse/BIGTOP-1047
          Hide
          cos Konstantin Boudnik added a comment -

          this doesn't looks like a very active JIRA. Besides, Whirr isn't look particularly thriving, if you know I mean. Are we doing it still?

          Show
          cos Konstantin Boudnik added a comment - this doesn't looks like a very active JIRA. Besides, Whirr isn't look particularly thriving, if you know I mean. Are we doing it still?
          Hide
          rvs Roman Shaposhnik added a comment -

          I think this one could be closed. We still need to refactor our Puppet code for puppet3 and everything, but that's a different story (and a different JIRA).

          Show
          rvs Roman Shaposhnik added a comment - I think this one could be closed. We still need to refactor our Puppet code for puppet3 and everything, but that's a different story (and a different JIRA).
          Hide
          cos Konstantin Boudnik added a comment -

          Closing the issue then. Linked it with Puppet 3.x work

          Show
          cos Konstantin Boudnik added a comment - Closing the issue then. Linked it with Puppet 3.x work

            People

            • Assignee:
              rvs Roman Shaposhnik
              Reporter:
              rvs Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development