Bigtop
  1. Bigtop
  2. BIGTOP-313

Move puppet configuration out of manifest and into extlookup() CSV

    Details

      Description

      Previously, bigtop-deploy puppet could only be configured by editing puppet manifests, or through a small number of environment variables that set facter facts. I've rewritten cluster.pp to use extlookup to read most configuration options from a site-specific CSV file with sane defaults for most options (only hadoop_head_node and bigtop_yumrepo_uri are required to be set). These changes should make it easier to automate deploying multiple clusters with different configurations from the same manifests.

      As written, the CSV files must live in $confdir/config/, where $confdir is the puppet config directory. This means that if you are manually calling 'puppet agent' on these manifests (rather than serving them from a puppet master), you are now required to specify --confdir in addition to --modulepath.

      At some point in the future, may be worth moving to hiera, as yaml is a lot prettier and more flexible than CSV. I did this using extlookup so as not to introduce external dependencies (since extlookup is now in puppet core).

      1. patch.txt
        17 kB
        Patrick Taylor Ramsey

        Activity

        Hide
        Roman Shaposhnik added a comment -

        Well, if you look at the suggested way of executing puppet you'll see that it expects configs to be in /etc/puppet/config. That makes our bigtop-deploy/puppet/config just a place for sample configs.

        A different way to look at it, of course, is --confdir and then it doesn't matter where you place your configs, except for the fact that Puppet (at least 2.x) expects your actual configuration to be under
        $CONFDIR/config. IOW, when you say something like --confdir=bigtop-deploy/puppet you still need that extra config folder at the end.

        Show
        Roman Shaposhnik added a comment - Well, if you look at the suggested way of executing puppet you'll see that it expects configs to be in /etc/puppet/config. That makes our bigtop-deploy/puppet/config just a place for sample configs. A different way to look at it, of course, is --confdir and then it doesn't matter where you place your configs, except for the fact that Puppet (at least 2.x) expects your actual configuration to be under $CONFDIR/config. IOW, when you say something like --confdir=bigtop-deploy/puppet you still need that extra config folder at the end.
        Hide
        jay vyas added a comment -

        Some of the information in this patch needs to be transferred up to BIGTOP-1365, i think....

        if somone gets a chance, please clarify for me why the configuration files go in "config/" instead of just --confdir .

        Seen some other folks who had some struggles getting the right config option from googling around http://apachebigtop.pbworks.com/w/file/fetch/52304359/Puppet.pdf getting this working.

        Show
        jay vyas added a comment - Some of the information in this patch needs to be transferred up to BIGTOP-1365 , i think.... if somone gets a chance, please clarify for me why the configuration files go in "config/" instead of just --confdir . Seen some other folks who had some struggles getting the right config option from googling around http://apachebigtop.pbworks.com/w/file/fetch/52304359/Puppet.pdf getting this working.
        Hide
        Roman Shaposhnik added a comment -

        Committed to both trunk and hadoop-0.23 branch

        Show
        Roman Shaposhnik added a comment - Committed to both trunk and hadoop-0.23 branch
        Hide
        Roman Shaposhnik added a comment -

        Patrick, the patch looks great. It cleans up areas that have been in dire need of a clean up – thanks a million for doing this!

        +1

        Show
        Roman Shaposhnik added a comment - Patrick, the patch looks great. It cleans up areas that have been in dire need of a clean up – thanks a million for doing this! +1
        Hide
        Patrick Taylor Ramsey added a comment -

        Updated patch.

        Show
        Patrick Taylor Ramsey added a comment - Updated patch.
        Hide
        Patrick Taylor Ramsey added a comment -

        Took down patch.txt while I fix the license headers and a few other issues.

        Show
        Patrick Taylor Ramsey added a comment - Took down patch.txt while I fix the license headers and a few other issues.
        Hide
        Bruno Mahé added a comment -

        The copyright headers are wrong.

        Show
        Bruno Mahé added a comment - The copyright headers are wrong.

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Patrick Taylor Ramsey
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development