Description
This ticket is about changing our Jenkins setup to do builds inside of containers so that we can auto-configure slaves and cleanup our environment as well as match containers that developers can use to build Bigtop.
I recommend using the Jenkins Mesos plugin (https://github.com/jenkinsci/mesos-plugin) to start Jenkins slaves, orchestrate and run jobs in containers, and handle resource allocation to different slave containers that start up. Apache Mesos handles the launching of containers and limiting containers resources so that slaves won’t starve each other out for resources while the Jenkins Mesos plugin sits on top of Mesos and starts the slave.jar inside of the container which will then run the Jenkins building the BigTop package you want or doing whatever tests you need in the isolated environment. It would also not require much changing of our current Jenkins setup to get it integrated and build steps for most projects would be unchanged. You would just install mesos-master somewhere and then install mesos-slave on all of the servers that currently do job builds, and Mesos will handle using them all and spinning up the Jenkins slaves.
While the above allows you to do BigTop builds in containers using Jenkins, we should also do some more things to make life easier. We can use the Jenkins Mesos plugin to run Docker in Docker (dind) so that inside of our Jenkins slave containers we can launch a Docker container inside of that to build Docker images for deployment. For example, you could have a Jenkins job that is kicked off inside of a container that then starts and builds a base Docker image that will later build a specific package on another job. This would allow our Jenkins setup to build and test our Puppet toolchain code and then produce images that would be used by other jobs for builds. This is documented and talked a lot about here: http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.U7R4t3VdVhF
As for matching builds on peoples computers, that gets pretty easy with Docker. You can just take the Jenkins Slave docker image and then run jobs against it without starting the slave.jar. So instead of docker run supervisord which you would do to get the Jenkins slave running, you can do docker run make hive-deb or whatever package you wanted. This would make peoples environments match our Jenkins environment exactly.
I have an experimental git repo with some Dockerfiles that are a pretty good prototype of this future setup located at https://github.com/jeid64/bigtop-dockerfiles jenkins-docker/ has an example Ubuntu 12.04 Dockerfile that once built uses the Puppet manifests in bigtop_toolchain to setup the build environment. When run by Jenkins, Jenkins will start a slave inside of the container and the slave would handle all the build steps for that build. That image can also just be used on your desktop without needing a Jenkins master or using anything from Jenkins.
Also in that repo is jenkins-dind/ which is based off of https://github.com/ahunnargikar/jenkins-dind multi-docker setup. This will be the image for dind image builds in Jenkins for build environments.
I would love to head up this ticket and work on getting everything setup.
Attachments
Issue Links
- is blocked by
-
BIGTOP-1568 bigtop01 needs gradle
-
- Resolved
-