Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2103

[Docker] Move bigtop/slaves image build to gradle

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: build, docker
    • Labels:
      None

      Description

      Slaves images are prefixed with trunk, or 1.1.0 for labeling, hence it'd be better to be able to specify prefix on build. We can solve this problem using gradle, which brings in a better UX.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user evans-ye opened a pull request:

          https://github.com/apache/bigtop/pull/45

          BIGTOP-2103. [Docker] Move bigtop/slaves image build to gradle and do…

          …wnload required packages during build

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/evans-ye/bigtop BIGTOP-2103

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/45.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #45


          commit ccd5aca12e0f0350b283bf473858a05937810ae4
          Author: Evans Ye <evansye@apache.org>
          Date: 2015-10-29T18:22:36Z

          BIGTOP-2103. [Docker] Move bigtop/slaves image build to gradle and download required packages during build


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user evans-ye opened a pull request: https://github.com/apache/bigtop/pull/45 BIGTOP-2103 . [Docker] Move bigtop/slaves image build to gradle and do… …wnload required packages during build You can merge this pull request into a Git repository by running: $ git pull https://github.com/evans-ye/bigtop BIGTOP-2103 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/45.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #45 commit ccd5aca12e0f0350b283bf473858a05937810ae4 Author: Evans Ye <evansye@apache.org> Date: 2015-10-29T18:22:36Z BIGTOP-2103 . [Docker] Move bigtop/slaves image build to gradle and download required packages during build
          Hide
          evans_ye Evans Ye added a comment -

          PR created.

          With this the images are built from gradle:

          Docker build tasks
          ------------------
          bigtop-slaves - Build bigtop/slaves images.
          Usage:
            $ ./gradlew -POS=[centos-6|centos-7|fedora-20|debian-8|ubuntu-14.04|opensuse-13.2] -Pprefix=STRING_TO_PREFIX bigtop-slaves
          Example:
            $ ./gradlew -POS=debian-8 -Pprefix=1.0.0 bigtop-slaves
          The built image name: bigtop/slaves:1.0.0-debian-8
          
          Show
          evans_ye Evans Ye added a comment - PR created. With this the images are built from gradle: Docker build tasks ------------------ bigtop-slaves - Build bigtop/slaves images. Usage: $ ./gradlew -POS=[centos-6|centos-7|fedora-20|debian-8|ubuntu-14.04|opensuse-13.2] -Pprefix=STRING_TO_PREFIX bigtop-slaves Example: $ ./gradlew -POS=debian-8 -Pprefix=1.0.0 bigtop-slaves The built image name: bigtop/slaves:1.0.0-debian-8
          Hide
          evans_ye Evans Ye added a comment -

          BTW Olaf Flebbe just a notice for you if you'd like to comment something, otherwise I'll commit this directly.

          Show
          evans_ye Evans Ye added a comment - BTW Olaf Flebbe just a notice for you if you'd like to comment something, otherwise I'll commit this directly.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Thanks Evans Ye for this patch. I am very busy and too slow to catch up ...

          The patch looks good to me, but I am concerned about copying to whole bigtop repository into the docker image
          COPY . /bigtop. I would prefer only to copy the bigtop_toolchain part to the right position in filesystem.

          Second I am missing the puppetize.sh call, or do I have missed something?

          Show
          oflebbe Olaf Flebbe added a comment - Thanks Evans Ye for this patch. I am very busy and too slow to catch up ... The patch looks good to me, but I am concerned about copying to whole bigtop repository into the docker image COPY . /bigtop . I would prefer only to copy the bigtop_toolchain part to the right position in filesystem. Second I am missing the puppetize.sh call, or do I have missed something?
          Hide
          evans_ye Evans Ye added a comment -

          Hey Olaf Flebbe thanks for the comment.
          That was my concern as well, but I ended up made the choice to copy the repository into image for the following reason:

          • For Bigtop users who'd like to try and build packages, a self-contained image comes along with bigtop repo would be nicer for them.
          • In order to cache gradle packages in side images, many separated pieces of code needs to be copied into container. This is very hard to maintain, and might be broken at some point our code changed.
          • We're building images ranged from 1 to 2GB in size, and to me this is the evil we must take, hence an extra 70MB repo is just making a slightly impact to the one-time downloading experience.

          I'd be happy to take your opinions. Probably there's a better way I didn't think of.

          Show
          evans_ye Evans Ye added a comment - Hey Olaf Flebbe thanks for the comment. That was my concern as well, but I ended up made the choice to copy the repository into image for the following reason: For Bigtop users who'd like to try and build packages, a self-contained image comes along with bigtop repo would be nicer for them. In order to cache gradle packages in side images, many separated pieces of code needs to be copied into container. This is very hard to maintain, and might be broken at some point our code changed. We're building images ranged from 1 to 2GB in size, and to me this is the evil we must take, hence an extra 70MB repo is just making a slightly impact to the one-time downloading experience. I'd be happy to take your opinions. Probably there's a better way I didn't think of.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Hi Evans Ye :

          I am not sure what you are trying to accomplish:

          Is it that you like to save the image after a build ? Than all the files in .m2/repository and are already there and rebuild will not download all the software artifacts again?

          In other words: Do you want to offer a full snapshot ready for bigtop-deploy to the world, where almost everything from source, dependencies to bigtop packages is contained within ?

          And extending the idea ... creating a docker image with everything contained with only one gradle call.

          • Building the bigtop-slave image (maybe pushing to dockerhub)
          • Running the docker run "gradle apt/yum" command on the created image
          • saving the result as bigtop-all and pushing to dockerhub

          That may be nice for students.

          I am not sure if I like it. Let me think about it.

          If I misunderstood you, please elaborate in more detail.

          Show
          oflebbe Olaf Flebbe added a comment - Hi Evans Ye : I am not sure what you are trying to accomplish: Is it that you like to save the image after a build ? Than all the files in .m2/repository and are already there and rebuild will not download all the software artifacts again? In other words: Do you want to offer a full snapshot ready for bigtop-deploy to the world, where almost everything from source, dependencies to bigtop packages is contained within ? And extending the idea ... creating a docker image with everything contained with only one gradle call. Building the bigtop-slave image (maybe pushing to dockerhub) Running the docker run "gradle apt/yum" command on the created image saving the result as bigtop-all and pushing to dockerhub That may be nice for students. I am not sure if I like it. Let me think about it. If I misunderstood you, please elaborate in more detail.
          Hide
          evans_ye Evans Ye added a comment -

          Olaf Flebbe sorry I didn't make it clear.
          I'm not going to build packages and save them into images.
          What I'm trying to do is just to save packages downloaded by gradle(~/.gradle) into images, so that we won't suffer unavailable issue on ci builds. And users won't download packages for gradle every time starting a new docker build slave.
          Those .m2 packages are not stored in images since they vary on different components, and required packages will be different if user changes BOM.

          puppetize.sh is done in bigtop/puppet, hence we don't need to do that again.

          Thanks for your quick respond anyway.

          Show
          evans_ye Evans Ye added a comment - Olaf Flebbe sorry I didn't make it clear. I'm not going to build packages and save them into images. What I'm trying to do is just to save packages downloaded by gradle(~/.gradle) into images, so that we won't suffer unavailable issue on ci builds . And users won't download packages for gradle every time starting a new docker build slave. Those .m2 packages are not stored in images since they vary on different components, and required packages will be different if user changes BOM. puppetize.sh is done in bigtop/puppet, hence we don't need to do that again. Thanks for your quick respond anyway.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Evans Ye I missed the (cd /bigtop; ./gradlew) part, now I understand.

          I propose this kind of Dockerfile:

          COPY bigtop_toolchain /etc/puppet/modules/
          COPY *.gradle /tmp
          RUN puppet apply -e "include bigtop_toolchain::installer" && (cd /tmp; ./gradlew)
          

          Thats a lot cleaner and more easy to understand. Copying everything, we will have a stale copy of bigtop hanging around in the image, potentially confusing users (like me).

          Show
          oflebbe Olaf Flebbe added a comment - Evans Ye I missed the (cd /bigtop; ./gradlew) part, now I understand. I propose this kind of Dockerfile: COPY bigtop_toolchain /etc/puppet/modules/ COPY *.gradle /tmp RUN puppet apply -e "include bigtop_toolchain::installer" && (cd /tmp; ./gradlew) Thats a lot cleaner and more easy to understand. Copying everything, we will have a stale copy of bigtop hanging around in the image, potentially confusing users (like me).
          Hide
          evans_ye Evans Ye added a comment -

          That's another way I tried earlier, but gradle and buildSrc folders, and a pom.xml deep inside bigtop-tests are also required by gradlew to successfully executed. That's why I wrote the following point:

          • In order to cache gradle packages in side images, many separated pieces of code needs to be copied into container. This is very hard to maintain, and might be broken at some point our code changed.

          To this end, do you think it's fine to list out all the required file and put them into /tmp for ./gradlew to execute? I think it's hard to judge though.

          Show
          evans_ye Evans Ye added a comment - That's another way I tried earlier, but gradle and buildSrc folders, and a pom.xml deep inside bigtop-tests are also required by gradlew to successfully executed. That's why I wrote the following point: In order to cache gradle packages in side images, many separated pieces of code needs to be copied into container. This is very hard to maintain, and might be broken at some point our code changed. To this end, do you think it's fine to list out all the required file and put them into /tmp for ./gradlew to execute? I think it's hard to judge though.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Argh you are right. There are a lot more dependencies.

          But please let's seperate the issues:

          1) a gradle task to create docker images, removing the reduant build.sh scripts

          2) optimize caching of downloads by gradle

          I am a huge fan of (1) . I support caching, but I do not like the consequences of the proposed patch, at least not for now. I would like to investigate the side images further, rather than have a ugly workaround now.

          Can we reword this JIRA to "Move bigtop/slaves image build to gradle " and file a new JIRA where we investigate other possible solutions?

          Show
          oflebbe Olaf Flebbe added a comment - Argh you are right. There are a lot more dependencies. But please let's seperate the issues: 1) a gradle task to create docker images, removing the reduant build.sh scripts 2) optimize caching of downloads by gradle I am a huge fan of (1) . I support caching, but I do not like the consequences of the proposed patch, at least not for now. I would like to investigate the side images further, rather than have a ugly workaround now. Can we reword this JIRA to "Move bigtop/slaves image build to gradle " and file a new JIRA where we investigate other possible solutions?
          Hide
          oflebbe Olaf Flebbe added a comment -

          Argh you are right. There are a lot more dependencies.

          But please let's seperate the issues:

          1) a gradle task to create docker images, removing the reduant build.sh scripts

          2) optimize caching of downloads by gradle

          I am a huge fan of (1) . I support caching, but I do not like the consequences of the proposed patch, at least not for now. I would like to investigate the side images further, rather than have a ugly workaround now.

          Can we reword this JIRA to "Move bigtop/slaves image build to gradle " and file a new JIRA where we investigate other possible solutions?

          Show
          oflebbe Olaf Flebbe added a comment - Argh you are right. There are a lot more dependencies. But please let's seperate the issues: 1) a gradle task to create docker images, removing the reduant build.sh scripts 2) optimize caching of downloads by gradle I am a huge fan of (1) . I support caching, but I do not like the consequences of the proposed patch, at least not for now. I would like to investigate the side images further, rather than have a ugly workaround now. Can we reword this JIRA to "Move bigtop/slaves image build to gradle " and file a new JIRA where we investigate other possible solutions?
          Hide
          evans_ye Evans Ye added a comment -

          No problem I can do that. Let me fire another JIRA.

          Show
          evans_ye Evans Ye added a comment - No problem I can do that. Let me fire another JIRA.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user evans-ye closed the pull request at:

          https://github.com/apache/bigtop/pull/45

          Show
          githubbot ASF GitHub Bot added a comment - Github user evans-ye closed the pull request at: https://github.com/apache/bigtop/pull/45
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user evans-ye opened a pull request:

          https://github.com/apache/bigtop/pull/47

          BIGTOP-2103. [Docker] Move bigtop/slaves image build to gradle

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/evans-ye/bigtop BIGTOP-2103

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/47.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #47


          commit 955422102eb323d0b448bc0acb8eeec0823df5cd
          Author: Evans Ye <evansye@apache.org>
          Date: 2015-11-01T16:07:28Z

          BIGTOP-2103. [Docker] Move bigtop/slaves image build to gradle


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user evans-ye opened a pull request: https://github.com/apache/bigtop/pull/47 BIGTOP-2103 . [Docker] Move bigtop/slaves image build to gradle You can merge this pull request into a Git repository by running: $ git pull https://github.com/evans-ye/bigtop BIGTOP-2103 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/47.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #47 commit 955422102eb323d0b448bc0acb8eeec0823df5cd Author: Evans Ye <evansye@apache.org> Date: 2015-11-01T16:07:28Z BIGTOP-2103 . [Docker] Move bigtop/slaves image build to gradle
          Hide
          evans_ye Evans Ye added a comment -

          centos-6 and debian-8 are tested.

          Show
          evans_ye Evans Ye added a comment - centos-6 and debian-8 are tested.
          Hide
          oflebbe Olaf Flebbe added a comment - - edited

          Seems to run for me too

          +1

          Will commit it in a few minutes

          Show
          oflebbe Olaf Flebbe added a comment - - edited Seems to run for me too +1 Will commit it in a few minutes
          Hide
          oflebbe Olaf Flebbe added a comment -

          Thanks Evans Ye ! Hm, I was not able to signoff a pull request ...

          Show
          oflebbe Olaf Flebbe added a comment - Thanks Evans Ye ! Hm, I was not able to signoff a pull request ...

            People

            • Assignee:
              evans_ye Evans Ye
              Reporter:
              evans_ye Evans Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development