Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1908

Move bigtop-deploy's dockerfiles into a centralized docker place

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: build, docker
    • Labels:
      None

      Description

      It would be great if we can have all the docker image building scripts consolidated into a centralized docker place. The docker image for bigtop docker provisioner can be built and pushed to dockerhub as well so that the docker provisioner can be easily consumed by bigtop users without requiring users to build image by themselves.

      1. BIGTOP-1908.patch
        17 kB
        Evans Ye
      2. BIGTOP-1908.patch
        14 kB
        Evans Ye
      3. BIGTOP-1908.patch
        17 kB
        Evans Ye
      4. BIGTOP-1908.patch
        17 kB
        Evans Ye
      5. BIGTOP-1908.patch
        15 kB
        Evans Ye
      6. BIGTOP-1908.patch
        15 kB
        Evans Ye
      7. BIGTOP-1908.patch
        14 kB
        Evans Ye

        Issue Links

          Activity

          Hide
          evans_ye Evans Ye added a comment -

          Attach a patch to:

          • refactor centos and debian dockerfile from bigtop-deploy/vm/vagrant-puppet-docker to docker/bigtop-deploy
          • remove original building scripts in bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh and related documents.
          Show
          evans_ye Evans Ye added a comment - Attach a patch to: refactor centos and debian dockerfile from bigtop-deploy/vm/vagrant-puppet-docker to docker/bigtop-deploy remove original building scripts in bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh and related documents.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Seems to work, mostly.

          We should remove puppet installation from
          bigtop-deploy/vm/utils/setup-env-centos.sh ?

          Show
          oflebbe Olaf Flebbe added a comment - Seems to work, mostly. We should remove puppet installation from bigtop-deploy/vm/utils/setup-env-centos.sh ?
          Hide
          evans_ye Evans Ye added a comment -

          Well, we can't do that since the VM provisioner(vagrant-puppet-vm) still needs that part of code. I can't find a set of promising vagrant boxes on vagrantcloud which already have puppet installed. The puppetlabs only provides CentOS 6, 7 and Ubuntu 14, while Debian, Fedora, and OpenSUSE are missing. I don't have a pretty solution here, but I think it's still fine since within that code snippet we do yum install only, which is an idempotent operation. And the pre-installed puppet still speeds up the deployment a little bit.

          Show
          evans_ye Evans Ye added a comment - Well, we can't do that since the VM provisioner(vagrant-puppet-vm) still needs that part of code. I can't find a set of promising vagrant boxes on vagrantcloud which already have puppet installed. The puppetlabs only provides CentOS 6, 7 and Ubuntu 14, while Debian, Fedora, and OpenSUSE are missing. I don't have a pretty solution here, but I think it's still fine since within that code snippet we do yum install only, which is an idempotent operation. And the pre-installed puppet still speeds up the deployment a little bit.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Sorry, this is not pretty.

          • The Vagrantfile (boot2docker part) still requires the Dockerfiles, we need to change this.
          • After a little bit thinking I would like to have puppet_stdlib and apt modules and rng-tools already activated in bigtop/puppet, since I need the for bigtop/slaves (openjdk8) BIGTOP-1936 as well and it is important to have
          • For the puppet-vagrant-docker provisionier the setup-env is not needed any more, since everything is already included in the image. We can remove it from docker-hadoop.sh. We still need code for local repo , but this isn't needed for other provisioners.
          • After a little more thinking, the Dockerfiles should be equivalent to the setup-env-distro files. We should reuse them in the Dockerfile.
          • For puppet-vagrant-vm the file setup-env can stay as is.
          Show
          oflebbe Olaf Flebbe added a comment - Sorry, this is not pretty. The Vagrantfile (boot2docker part) still requires the Dockerfiles, we need to change this. After a little bit thinking I would like to have puppet_stdlib and apt modules and rng-tools already activated in bigtop/puppet, since I need the for bigtop/slaves (openjdk8) BIGTOP-1936 as well and it is important to have For the puppet-vagrant-docker provisionier the setup-env is not needed any more, since everything is already included in the image. We can remove it from docker-hadoop.sh. We still need code for local repo , but this isn't needed for other provisioners. After a little more thinking, the Dockerfiles should be equivalent to the setup-env-distro files. We should reuse them in the Dockerfile. For puppet-vagrant-vm the file setup-env can stay as is.
          Hide
          evans_ye Evans Ye added a comment - - edited
          • Oops, should remove the image section in Vagrantfile, sorry.
          • That's an option. What you need in BIGTOP-1936 is the apt module. If we do it in this way, we need to separate out the rng-tool, enable local repo, and mkdir code snippets, otherwise our bigtop/puppet images will be tainted.
          • Follow the logic above, I think its the time to refactor and isolate out puppet and puppet module installation code from setup-env*.sh. We can put that code in bigtop_toolchain/bin/puppetize.sh so that we can reuse it anywhere. After all bigtop_toolchain needs someone to setup puppet for it, hence providing puppetize.sh as its init script is natural. By having puppetize.sh we can better support to setup bare-metal, VM build slaves as well.
          • In bigtop/puppet's Dockerfiles, execute puppetize.sh and ./gradlew toolchain-puppetmodules to make iimages puppetized.
          • For Docker provisioner, since we already separated out puppet code from setup-env*.sh, there should be no code change.
          • For VM provisioner, executing puppetize.sh script must be added in its Vagrantfile.

          Any suggestion to make it better?

          Show
          evans_ye Evans Ye added a comment - - edited Oops, should remove the image section in Vagrantfile, sorry. That's an option. What you need in BIGTOP-1936 is the apt module. If we do it in this way, we need to separate out the rng-tool, enable local repo, and mkdir code snippets, otherwise our bigtop/puppet images will be tainted. Follow the logic above, I think its the time to refactor and isolate out puppet and puppet module installation code from setup-env*.sh. We can put that code in bigtop_toolchain/bin/puppetize.sh so that we can reuse it anywhere. After all bigtop_toolchain needs someone to setup puppet for it, hence providing puppetize.sh as its init script is natural. By having puppetize.sh we can better support to setup bare-metal, VM build slaves as well. In bigtop/puppet's Dockerfiles, execute puppetize.sh and ./gradlew toolchain-puppetmodules to make iimages puppetized. For Docker provisioner, since we already separated out puppet code from setup-env*.sh, there should be no code change. For VM provisioner, executing puppetize.sh script must be added in its Vagrantfile. Any suggestion to make it better?
          Hide
          oflebbe Olaf Flebbe added a comment -

          This iteration sounds very good:

          I agree to you second point.

          3rd point: I like the name puppetize.sh and agree to you plan.

          4rd: Yep

          5th: I do not fully understand, maybe code makes it more transparent your point to me.

          6th: Yep.

          Would you make a proposal for a reworked patch for all this, or should we seperate out parts of it, since this is a rather large change?

          Show
          oflebbe Olaf Flebbe added a comment - This iteration sounds very good: I agree to you second point. 3rd point: I like the name puppetize.sh and agree to you plan. 4rd: Yep 5th: I do not fully understand, maybe code makes it more transparent your point to me. 6th: Yep. Would you make a proposal for a reworked patch for all this, or should we seperate out parts of it, since this is a rather large change?
          Hide
          evans_ye Evans Ye added a comment -

          5th: In the design we delete puppetize code in setup-env*.sh, then we add puppetize code into bigtop/deploy images. So in the end, it all looks the same for Docker provisioner, i.e. there's no code change involved in docker-hadoop.sh.

          Thanks for your review on the design, I think it's better to separate out parts so that we can do pair review. We can leverage some exist JIRAs. How about we create an Umbrella JIRA and convert those related as sub-tasks? Now we have BIGTOP-1908, BIGTOP-2017, we might need to add one for puppetize, one for executing puppetize.sh in VM provisioner. I don't mind to take want part. I can either do the dev or review.

          Show
          evans_ye Evans Ye added a comment - 5th: In the design we delete puppetize code in setup-env*.sh, then we add puppetize code into bigtop/deploy images. So in the end, it all looks the same for Docker provisioner, i.e. there's no code change involved in docker-hadoop.sh. Thanks for your review on the design, I think it's better to separate out parts so that we can do pair review. We can leverage some exist JIRAs. How about we create an Umbrella JIRA and convert those related as sub-tasks? Now we have BIGTOP-1908 , BIGTOP-2017 , we might need to add one for puppetize, one for executing puppetize.sh in VM provisioner. I don't mind to take want part. I can either do the dev or review.
          Hide
          oflebbe Olaf Flebbe added a comment -

          As a first step I would like to create a new JIRA for refactor puppetizing out if bigtop-puppet containers.

          Additionaly, I would rework BIGTOP-2017 based on this.

          If you can rebase the work of this JIRA on top of step 1, that would be awesome.

          I am not so familiar with the VM provisioner ...I would prefer reviewing this.

          Show
          oflebbe Olaf Flebbe added a comment - As a first step I would like to create a new JIRA for refactor puppetizing out if bigtop-puppet containers. Additionaly, I would rework BIGTOP-2017 based on this. If you can rebase the work of this JIRA on top of step 1, that would be awesome. I am not so familiar with the VM provisioner ...I would prefer reviewing this.
          Hide
          evans_ye Evans Ye added a comment -

          Sure. I can do that. It's late here in my timezone. I'll follow up with you tomorrow.

          Show
          evans_ye Evans Ye added a comment - Sure. I can do that. It's late here in my timezone. I'll follow up with you tomorrow.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Sure. It's getting late for me now, too..

          Show
          oflebbe Olaf Flebbe added a comment - Sure. It's getting late for me now, too..
          Hide
          evans_ye Evans Ye added a comment -

          New patch uploaded. List of items changed in the patch:

          • Remove image building in README since we provide them directly on dockerhub.
          • Remove image building code snippets in Vagrantfile and docker-hadoop.sh
          • Move centos-6 and debian-8 Dockerfiles into bigtop/docker/bigtop-deploy
          • Adopt new bigtop/puppet as base images to build bigtop-deploy images

          Please review.

          Show
          evans_ye Evans Ye added a comment - New patch uploaded. List of items changed in the patch: Remove image building in README since we provide them directly on dockerhub. Remove image building code snippets in Vagrantfile and docker-hadoop.sh Move centos-6 and debian-8 Dockerfiles into bigtop/docker/bigtop-deploy Adopt new bigtop/puppet as base images to build bigtop-deploy images Please review.
          Hide
          evans_ye Evans Ye added a comment -

          New patch uploaded.
          The items modified in the patch is the same as my comment above but will additional improvement in destroy function in docker-hadoop.sh, which I use vagrant destroy -f directly because there's no need to save image container anymore.

          Show
          evans_ye Evans Ye added a comment - New patch uploaded. The items modified in the patch is the same as my comment above but will additional improvement in destroy function in docker-hadoop.sh , which I use vagrant destroy -f directly because there's no need to save image container anymore.
          Hide
          cos Konstantin Boudnik added a comment -

          Any reason there's no support for ubuntu, but just debian?

          Show
          cos Konstantin Boudnik added a comment - Any reason there's no support for ubuntu, but just debian?
          Hide
          oflebbe Olaf Flebbe added a comment -

          That's BIGTOP-2015

          Show
          oflebbe Olaf Flebbe added a comment - That's BIGTOP-2015
          Hide
          cos Konstantin Boudnik added a comment -

          That's why linking issues together is such a good idea Thanks!

          Show
          cos Konstantin Boudnik added a comment - That's why linking issues together is such a good idea Thanks!
          Hide
          oflebbe Olaf Flebbe added a comment -

          The issue at hand has nothing todo with support for deployment for ubuntu.

          Show
          oflebbe Olaf Flebbe added a comment - The issue at hand has nothing todo with support for deployment for ubuntu.
          Hide
          evans_ye Evans Ye added a comment -

          The reason there's no ubuntu is just because we haven't add it
          If we get this in then we can get back to BIGTOP-2015.

          Show
          evans_ye Evans Ye added a comment - The reason there's no ubuntu is just because we haven't add it If we get this in then we can get back to BIGTOP-2015 .
          Hide
          evans_ye Evans Ye added a comment -

          New patch fix the license issue.

          Show
          evans_ye Evans Ye added a comment - New patch fix the license issue.
          Hide
          cos Konstantin Boudnik added a comment -

          Yup, got it - thanks!

          Show
          cos Konstantin Boudnik added a comment - Yup, got it - thanks!
          Hide
          evans_ye Evans Ye added a comment -

          Rebase on the master and tested !
          Now, the newest patch should be good for review.

          Show
          evans_ye Evans Ye added a comment - Rebase on the master and tested ! Now, the newest patch should be good for review.
          Hide
          evans_ye Evans Ye added a comment -

          I've addressed the issues. If no more comments, I'll commit this after my local test passed.

          Show
          evans_ye Evans Ye added a comment - I've addressed the issues. If no more comments, I'll commit this after my local test passed.
          Hide
          evans_ye Evans Ye added a comment -

          Committed and pushed according to CTR.

          Show
          evans_ye Evans Ye added a comment - Committed and pushed according to CTR.

            People

            • Assignee:
              evans_ye Evans Ye
              Reporter:
              evans_ye Evans Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development