Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: General
    • Labels:
      None

      Description

      Summary of the bigtop-dev discussion:

      A few approaches have been suggested:

      1. (Roman) maintain a parallel (very shallow) collection of puppet code that would, essentially, manage our "build slaves"
      2. (Roman) do #1 but automate it in such a way that the info actually gets harvested from spec/conrol files. (Cos suggested to use Groovy for implementation)
      3. (Bruno) VMs.

      There are some pros and cons for each of them.

      For example:

      #3 Pros:
      (Bruno) Tools like Boxgrinder and Oz can deal with multiple OSes and can create local images as well as push them to the cloud. The build would be repeatable and would not require any effort from the end user (apart maybe providing Oracle JDK, but that would have to be the case whatever the solution). Future contributors would just need to boot their VM to get started and hopefully ease contribution.
      (Roman) I'd love if I could say

      $ vagrant add box/up
      

      and get the environment up and running without much fuzz.

      #3 Cons:
      (Cos) I think boxed environments (like VMs) are an overkill... old proven toolchain type of environment that can automatically bootstrap upon a fresh install (or update itself: think Maven model) and pull whatever apps are required in whatever form they might exist... Such approach would be more fluid than a somewhat rigid VMs, that would have to be updated periodically, versioned, etc. Another benefit of a toolchain is that BigTop packages might have to redistribute/wrap some of the tools for later use once a package is installed on a customer's system.

      There are some tools based on SRPMs that can extract a dependency list.

      Roman suggested a simpler approach of grepping and using sed/awk:

      $ git grep -E 'Build(Requires|-Depends):'
      

      Sean was ... in favor of the idea of extracting dependencies from the control and spec files and building a script that will install the necessary tool chain.

      Cos mentioned ... It should work, except for build-time dependencies which aren't declared in the bigtop packages (e.g. libtool, etc.). So, I'd vote for a separated list of build-time dependencies being maintained.

      Roman separated 3 distinct use cases:

      1. provisioning/maintaining a long running Jenkins slave
      2. provisioning a host dev environment
      3. provisioning a VM dev environment

      Roman also suggested ... a 'virtual' package called build so that running:

      $ make build-[rpm|deb]
      

      will create a package that would download all the bits and pieced of a tool-chain AND also promote BuildRequires: to its own Requires: so that installing this package will give you all the toolchain bits and all the package deps at the same time.

      The consensus seems to be that all approaches are valid for some use cases, so implementation may start from simpler approaches, and evolve as necessary.

        Issue Links

          Activity

          Roman Shaposhnik made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Konstantin Boudnik added a comment -

          Sorry, my bad - I misread the code.
          WRT ubuntu: I have opened BIGTOP-927 - feel free to grab it!

          Show
          Konstantin Boudnik added a comment - Sorry, my bad - I misread the code. WRT ubuntu: I have opened BIGTOP-927 - feel free to grab it!
          Hide
          Ian Mordey added a comment -

          It should work. The modules/bigtop-toolchain/mrdocs-protobuf-rpm.repo is picked up from bigtop-toolchain/files. As far as I can see this exists in the correct place:
          bigtop/bigtop-toolchain/files/mrdocs-protobuf-rpm.repo

          BTW I've done some work on the Ubuntu version. Is there another Jira for this? I'll probably have a patch ready by the weekend.

          Cheers

          Show
          Ian Mordey added a comment - It should work. The modules/bigtop-toolchain/mrdocs-protobuf-rpm.repo is picked up from bigtop-toolchain/files. As far as I can see this exists in the correct place: bigtop/bigtop-toolchain/files/mrdocs-protobuf-rpm.repo BTW I've done some work on the Ubuntu version. Is there another Jira for this? I'll probably have a patch ready by the weekend. Cheers
          Konstantin Boudnik made changes -
          Link This issue is related to BIGTOP-810 [ BIGTOP-810 ]
          Hide
          Konstantin Boudnik added a comment -

          BTW, Ian Mordey, it seems that
          class bigtop-toolchain::protobuf {

          won't work on centos as the
          puppet:///modules/bigtop-toolchain/mrdocs-protobuf-rpm.repo
          isn't present. Is it an omission?

          Show
          Konstantin Boudnik added a comment - BTW, Ian Mordey , it seems that class bigtop-toolchain::protobuf { won't work on centos as the puppet:///modules/bigtop-toolchain/mrdocs-protobuf-rpm.repo isn't present. Is it an omission?
          Konstantin Boudnik made changes -
          Link This issue is related to BIGTOP-927 [ BIGTOP-927 ]
          Konstantin Boudnik made changes -
          Assignee Konstantin Boudnik [ cos ] Ian Mordey [ imordey ]
          Konstantin Boudnik made changes -
          Assignee Konstantin Boudnik [ cos ]
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Assignee Anatoli Fomenko [ anatoli.fomenko ]
          Fix Version/s 0.6.0 [ 12323895 ]
          Resolution Fixed [ 1 ]
          Hide
          Konstantin Boudnik added a comment -

          I have just committed it to the trunk. Thanks Ian Mordey

          Show
          Konstantin Boudnik added a comment - I have just committed it to the trunk. Thanks Ian Mordey
          Hide
          Konstantin Boudnik added a comment -

          Now, it works out of the box once the files are added to the files.

          I would +1 on the patch in the current shape and open two other jiras:

          • implement automatic provisioning for files
          • cover Ubuntu[,Suse] platform

          I will commit it shortly unless there any objections

          Show
          Konstantin Boudnik added a comment - Now, it works out of the box once the files are added to the files . I would +1 on the patch in the current shape and open two other jiras: implement automatic provisioning for files cover Ubuntu [,Suse] platform I will commit it shortly unless there any objections
          Hide
          Ian Mordey added a comment -

          There's a couple of issues with getting it working on Debian/Ubuntu. The protobuf used in this module is pulled from a yum repo as suggested in BIGTOP-810. Although there are packages for Ubuntu they stop at 11.10. We'd either need to build protobuf from source or find a more up-to-date location for 2.4.1 deb files.

          And yes JDK. I have another module written specifically for JDK in Ubuntu that we use internally. I can factor this into the bigtop-toolchain module without much difficulty.

          Show
          Ian Mordey added a comment - There's a couple of issues with getting it working on Debian/Ubuntu. The protobuf used in this module is pulled from a yum repo as suggested in BIGTOP-810 . Although there are packages for Ubuntu they stop at 11.10. We'd either need to build protobuf from source or find a more up-to-date location for 2.4.1 deb files. And yes JDK. I have another module written specifically for JDK in Ubuntu that we use internally. I can factor this into the bigtop-toolchain module without much difficulty.
          Ian Mordey made changes -
          Hide
          Ian Mordey added a comment -

          Made the changes as Cos suggested.

          Show
          Ian Mordey added a comment - Made the changes as Cos suggested.
          Ian Mordey made changes -
          Attachment 0001-Add-bigtop-toolchain-and-puppet-manifests-for-deploy.patch [ 12576093 ]
          Hide
          Konstantin Boudnik added a comment -

          Great start, Ian. A couple of comments:

          • the patch delivers the recipes to bigtop-toolchain/puppet while classes definition expect module name to be puppet-bigtoptc. This renders the patch unusable unless % mv puppet puppet-bigtoptc first
          • I would suggest to call the module just bigtoptc, perhaps, or just bigtop-toolchain so it can be used directly from the toplevel directory of the Bigtop project
          • class forrest can be simplified to
            class puppet-bigtoptc::forrest{
            
              file{ '/tmp/apache-forrest-0.9.tar.gz':
                source => 'puppet:///modules/puppet-bigtoptc/apache-forrest-0.9.tar.gz',
                ensure => present,
                owner  => root,
                group  => root,
                mode   => 755
              }
            
              exec{'/bin/tar xvzf /tmp/apache-forrest-0.9.tar.gz':
                cwd         => '/usr/local',
                refreshonly => true,
                subscribe   => File["/tmp/apache-forrest-0.9.tar.gz"],
              }
            
              file { '/usr/local/apache-forrest':
                ensure  => link,
                target  => '/usr/local/apache-forrest-0.9',
                require => Exec['/bin/tar xvzf /tmp/apache-forrest-0.9.tar.gz'],
              }
            }
            

            if one use complete binary distribution from [here|http://archive.apache.org/dist/forrest/0.9/apache-forrest-0.9.tar.gz}}

          • I'd suggest to ensure the presence of needed tarballs/packages in files/ and download them if needed using the same puppet recipes, instead of stating the requirement for a user to download them first. The whole purpose of this exercise is to provide a 'single button push' dev. environment creation. I think this can be done as a separate JIRA; for now we can just update README file with the links to the tarballs. Something like:
            The following files must also be downloaded from their respective mirrors and copied into files/
            
            apache-ant-1.9.0-bin.tar.gz
              http://mirrors.ibiblio.org/apache//ant/binaries/apache-ant-1.9.0-bin.tar.gz
            
            apache-forrest-0.9-dependencies.tar.gz
              http://archive.apache.org/dist/forrest/0.9/apache-forrest-0.9.tar.gz
            
            apache-forrest-0.9-sources.tar.gz
            
            apache-maven-3.0.5-bin.tar.gz
              ftp://mirror.reverse.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
            
          • class jdk won't work on Ubuntu: it needs to have ability to install either rpm or deb package. This is one can be resolved later as a separate JIRA.
          Show
          Konstantin Boudnik added a comment - Great start, Ian. A couple of comments: the patch delivers the recipes to bigtop-toolchain/puppet while classes definition expect module name to be puppet-bigtoptc . This renders the patch unusable unless % mv puppet puppet-bigtoptc first I would suggest to call the module just bigtoptc , perhaps, or just bigtop-toolchain so it can be used directly from the toplevel directory of the Bigtop project class forrest can be simplified to class puppet-bigtoptc::forrest{ file{ '/tmp/apache-forrest-0.9.tar.gz': source => 'puppet:///modules/puppet-bigtoptc/apache-forrest-0.9.tar.gz', ensure => present, owner => root, group => root, mode => 755 } exec{'/bin/tar xvzf /tmp/apache-forrest-0.9.tar.gz': cwd => '/usr/local', refreshonly => true, subscribe => File["/tmp/apache-forrest-0.9.tar.gz"], } file { '/usr/local/apache-forrest': ensure => link, target => '/usr/local/apache-forrest-0.9', require => Exec['/bin/tar xvzf /tmp/apache-forrest-0.9.tar.gz'], } } if one use complete binary distribution from [here|http://archive.apache.org/dist/forrest/0.9/apache-forrest-0.9.tar.gz}} I'd suggest to ensure the presence of needed tarballs/packages in files/ and download them if needed using the same puppet recipes, instead of stating the requirement for a user to download them first. The whole purpose of this exercise is to provide a 'single button push' dev. environment creation. I think this can be done as a separate JIRA; for now we can just update README file with the links to the tarballs. Something like: The following files must also be downloaded from their respective mirrors and copied into files/ apache-ant-1.9.0-bin.tar.gz http://mirrors.ibiblio.org/apache//ant/binaries/apache-ant-1.9.0-bin.tar.gz apache-forrest-0.9-dependencies.tar.gz http://archive.apache.org/dist/forrest/0.9/apache-forrest-0.9.tar.gz apache-forrest-0.9-sources.tar.gz apache-maven-3.0.5-bin.tar.gz ftp://mirror.reverse.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz class jdk won't work on Ubuntu: it needs to have ability to install either rpm or deb package. This is one can be resolved later as a separate JIRA.
          Ian Mordey made changes -
          Attachment 0001-Add-bigtop-toolchain-and-puppet-manifests-for-deploy.patch [ 12576093 ]
          Ian Mordey made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Ian Mordey added a comment -

          Patch to add bigtop-toolchain/puppet folders containing my manfiests.

          Show
          Ian Mordey added a comment - Patch to add bigtop-toolchain/puppet folders containing my manfiests.
          Hide
          Konstantin Boudnik added a comment -

          Nope, sorry - I missed that part. Looks great, actually!

          Show
          Konstantin Boudnik added a comment - Nope, sorry - I missed that part. Looks great, actually!
          Hide
          Ian Mordey added a comment -

          Yes I can generate as a patch tomorrow.

          Cos. I already generate a file in /etc/profile.d that should set the following environment variables:
          export MAVEN_HOME=/usr/local/maven
          export PATH=$PATH:$MAVEN_HOME/bin
          export JAVA_HOME=/usr/java/latest
          export ANT_HOME=/usr/local/ant
          export PATH=$PATH:$ANT_HOME/bin
          export FORREST_HOME=/usr/local/apache-forrest
          export PATH=$PATH:$FORREST_HOME/bin

          Do you require any others setting?

          Show
          Ian Mordey added a comment - Yes I can generate as a patch tomorrow. Cos. I already generate a file in /etc/profile.d that should set the following environment variables: export MAVEN_HOME=/usr/local/maven export PATH=$PATH:$MAVEN_HOME/bin export JAVA_HOME=/usr/java/latest export ANT_HOME=/usr/local/ant export PATH=$PATH:$ANT_HOME/bin export FORREST_HOME=/usr/local/apache-forrest export PATH=$PATH:$FORREST_HOME/bin Do you require any others setting?
          Hide
          Konstantin Boudnik added a comment -

          I would also love to see a bash env. file being generated with all needed variables exported and PATH properly set!

          Show
          Konstantin Boudnik added a comment - I would also love to see a bash env. file being generated with all needed variables exported and PATH properly set!
          Hide
          Roman Shaposhnik added a comment -

          Ian, this looks like a really good start. Would it be possible for you to shape it into a patch for Bigtop?

          Show
          Roman Shaposhnik added a comment - Ian, this looks like a really good start. Would it be possible for you to shape it into a patch for Bigtop?
          Hide
          Ian Mordey added a comment -

          As Cos says I've written some puppet manifests to install the dependencies required for a bigtop RPM build. I tested them on a CentOS box this morning and could successfully make hadoop-rpm after running it. They're probably a bit sloppy as I'm fairly new to puppet so any improvements would be gratefully accepted! They're on GitHub here:
          https://github.com/imordey/puppet-bigtoptc

          Cheers

          Show
          Ian Mordey added a comment - As Cos says I've written some puppet manifests to install the dependencies required for a bigtop RPM build. I tested them on a CentOS box this morning and could successfully make hadoop-rpm after running it. They're probably a bit sloppy as I'm fairly new to puppet so any improvements would be gratefully accepted! They're on GitHub here: https://github.com/imordey/puppet-bigtoptc Cheers
          Konstantin Boudnik made changes -
          Link This issue is related to BIGTOP-810 [ BIGTOP-810 ]
          Konstantin Boudnik made changes -
          Link This issue is related to BIGTOP-810 [ BIGTOP-810 ]
          Hide
          Konstantin Boudnik added a comment -

          We've been using a set of Puppet recipes to manage slaves and configure dev. environments with all BigTop needed bits. It works pretty well with three quirks (that would hunt any solution anyway):

          • jdk
          • ant/mvn
          • forrest

          We simply keep the needed tarballs as Puppet's file resources and unpack them as needed. This can be changed toward the direct download on every installation (a matter of bandwidths preferences). The only real downside of Puppet's approach is that you'd need to have Puppet first, but it is available for any mainstream distro, I believe.

          Show
          Konstantin Boudnik added a comment - We've been using a set of Puppet recipes to manage slaves and configure dev. environments with all BigTop needed bits. It works pretty well with three quirks (that would hunt any solution anyway): jdk ant/mvn forrest We simply keep the needed tarballs as Puppet's file resources and unpack them as needed. This can be changed toward the direct download on every installation (a matter of bandwidths preferences). The only real downside of Puppet's approach is that you'd need to have Puppet first, but it is available for any mainstream distro, I believe.
          Roman Shaposhnik made changes -
          Fix Version/s 0.6.0 [ 12323895 ]
          Roman Shaposhnik made changes -
          Field Original Value New Value
          Fix Version/s 0.6.0 [ 12323895 ]
          Fix Version/s 0.5.0 [ 12321865 ]
          Hide
          Konstantin Boudnik added a comment -

          Thanks for putting this all together!

          Show
          Konstantin Boudnik added a comment - Thanks for putting this all together!
          Anatoli Fomenko created issue -

            People

            • Assignee:
              Ian Mordey
              Reporter:
              Anatoli Fomenko
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development