Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2110

[Docker] Cache packages required by gradle to execute into bigtop/slaves images

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: build, docker
    • Labels:
      None

      Description

      Our new ci build for packages now suffers an issue that sometimes gradle packages downloading can be failed due to us sending massive amount of requests. This can be solved by downloading required packages during image build time.

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          Just to keep the conversation in the same place: back in a day we had a discussion of how we can deploy/keep all artifacts produced inside of a Bigtop build at some local Maven repository, which could be a dockerized service or something. I believe Olaf has even published the configuration for his Artifactory (or similar) into a JIRA ticket, IIRC.

          Show
          cos Konstantin Boudnik added a comment - Just to keep the conversation in the same place: back in a day we had a discussion of how we can deploy/keep all artifacts produced inside of a Bigtop build at some local Maven repository, which could be a dockerized service or something. I believe Olaf has even published the configuration for his Artifactory (or similar) into a JIRA ticket, IIRC.
          Hide
          evans_ye Evans Ye added a comment -

          Yup it is an option. What I am concern about is user experience. Using nexus or artifactory can solve our build problem for CI, but for user who would like to use slaves images, they stil need to download gradle packages eveytime starting up a new container. Maybe we should get the nexus patch first in and leave me concern separated.

          Show
          evans_ye Evans Ye added a comment - Yup it is an option. What I am concern about is user experience. Using nexus or artifactory can solve our build problem for CI, but for user who would like to use slaves images, they stil need to download gradle packages eveytime starting up a new container. Maybe we should get the nexus patch first in and leave me concern separated.
          Hide
          cos Konstantin Boudnik added a comment -

          Good point, although I have never seen this issue in my experience. Ok, let me see if we can add the plugins into the image somehow...

          Show
          cos Konstantin Boudnik added a comment - Good point, although I have never seen this issue in my experience. Ok, let me see if we can add the plugins into the image somehow...
          Hide
          cos Konstantin Boudnik added a comment -

          Ok, one of the possible ways to solve this is to add a minimal required .gradle content into the slave images. From a little experiment, I can see it will add about 100MB (about 50MB is just the wrapper) to an image, but it will nullify the need to download all the plugins on every new build, saving the time and bandwidth.

          There a couple of issues with it:

          • the cache needs to be rebuild every time the slave image is built to make sure we aren't missing any important updates like version of the gradle being bumped up, or an explicit plugin is upgraded.
          • we need to figure out where to put this cache. It can set in to somewhere like /usr/share/gradle.home. Add the setting of GRADLE_USER_HOME pointing to that location to the /etc/profiles.d/bigtop.sh so the build is picking up the cache from the correct location. Alternatively, gradle can be run with -g option to achieve the same.

          Running something like gradle tasks is enough to refill the cache.

          I am not sure how to do the better in the docker creation process. Perhaps, we can do something like this:

          • gradle -g /tmp/gradle.shared
            Then inside of a docker file add
          • COPY /tmp/gradle.shared /usr/share

          The first step only need to be executed once for all docker image builds. Does it sound right? Olaf Flebbe, Evans Ye - any input?

          Show
          cos Konstantin Boudnik added a comment - Ok, one of the possible ways to solve this is to add a minimal required .gradle content into the slave images. From a little experiment, I can see it will add about 100MB (about 50MB is just the wrapper) to an image, but it will nullify the need to download all the plugins on every new build, saving the time and bandwidth. There a couple of issues with it: the cache needs to be rebuild every time the slave image is built to make sure we aren't missing any important updates like version of the gradle being bumped up, or an explicit plugin is upgraded. we need to figure out where to put this cache. It can set in to somewhere like /usr/share/gradle.home . Add the setting of GRADLE_USER_HOME pointing to that location to the /etc/profiles.d/bigtop.sh so the build is picking up the cache from the correct location. Alternatively, gradle can be run with -g option to achieve the same. Running something like gradle tasks is enough to refill the cache. I am not sure how to do the better in the docker creation process. Perhaps, we can do something like this: gradle -g /tmp/gradle.shared Then inside of a docker file add COPY /tmp/gradle.shared /usr/share The first step only need to be executed once for all docker image builds. Does it sound right? Olaf Flebbe , Evans Ye - any input?
          Hide
          evans_ye Evans Ye added a comment -

          Sounds like a good idea!
          We can implement the cache generation as gradle task and let the docker image creation depends on that task. In this way the cache only need to be generated once.

          Show
          evans_ye Evans Ye added a comment - Sounds like a good idea! We can implement the cache generation as gradle task and let the docker image creation depends on that task. In this way the cache only need to be generated once.
          Hide
          cos Konstantin Boudnik added a comment -

          I gave it a quick try, but it looks like running a gradle build from inside of the gradle build isn't going to work. It starts fine and I can see the content of the cache pulled in, but at the end it throws an NPE and terminates, cleaning the content of the new cache.
          Pretty weird...

          Show
          cos Konstantin Boudnik added a comment - I gave it a quick try, but it looks like running a gradle build from inside of the gradle build isn't going to work. It starts fine and I can see the content of the cache pulled in, but at the end it throws an NPE and terminates, cleaning the content of the new cache. Pretty weird...
          Hide
          evans_ye Evans Ye added a comment -

          Mine is working properly during the test. But I didn't change the GRADLE_USER_HOME. Have you?

          Show
          evans_ye Evans Ye added a comment - Mine is working properly during the test. But I didn't change the GRADLE_USER_HOME. Have you?
          Hide
          cos Konstantin Boudnik added a comment -

          I am using "-g" command line option for the spawn process. I want to guarantee that new cache doesn't have anything in it, but what's needed for initial gradle run.

          Show
          cos Konstantin Boudnik added a comment - I am using "-g" command line option for the spawn process. I want to guarantee that new cache doesn't have anything in it, but what's needed for initial gradle run.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user evans-ye opened a pull request:

          https://github.com/apache/bigtop/pull/59

          BIGTOP-2110. [Docker] Cache packages required by gradle to execute in…

          …to bigtop/slaves images

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/evans-ye/bigtop BIGTOP-2110

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/59.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #59


          commit 365a31d1fa9c415c5331d08e2a0ff1549c09543b
          Author: Evans Ye <evansye@apache.org>
          Date: 2015-12-02T16:18:35Z

          BIGTOP-2110. [Docker] Cache packages required by gradle to execute into bigtop/slaves images


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user evans-ye opened a pull request: https://github.com/apache/bigtop/pull/59 BIGTOP-2110 . [Docker] Cache packages required by gradle to execute in… …to bigtop/slaves images You can merge this pull request into a Git repository by running: $ git pull https://github.com/evans-ye/bigtop BIGTOP-2110 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/59.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #59 commit 365a31d1fa9c415c5331d08e2a0ff1549c09543b Author: Evans Ye <evansye@apache.org> Date: 2015-12-02T16:18:35Z BIGTOP-2110 . [Docker] Cache packages required by gradle to execute into bigtop/slaves images
          Hide
          evans_ye Evans Ye added a comment -

          I put the PR on because the code is better for thousand words.

          Docker images tested here: http://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-Trunk-Evans/
          Packages built by the cached docker images: http://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages-Evans/

          It seems things are working properly(if ignore debian's maven download issue during image build), although there're still some maven networking issues that fail the package builds, but that's another story.

          Show
          evans_ye Evans Ye added a comment - I put the PR on because the code is better for thousand words. Docker images tested here: http://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-Trunk-Evans/ Packages built by the cached docker images: http://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages-Evans/ It seems things are working properly(if ignore debian's maven download issue during image build), although there're still some maven networking issues that fail the package builds, but that's another story.
          Hide
          cos Konstantin Boudnik added a comment - - edited

          I have already replied on the PR - the change looks good and I am glad you've figured out how to make it work, where I got stuck. I can commit it, unless you're still up

          As for the maven downloads - this musta got fixed once we deploy the nexus proxy.

          Show
          cos Konstantin Boudnik added a comment - - edited I have already replied on the PR - the change looks good and I am glad you've figured out how to make it work, where I got stuck. I can commit it, unless you're still up As for the maven downloads - this musta got fixed once we deploy the nexus proxy.
          Hide
          evans_ye Evans Ye added a comment -

          Committed and pushed.
          Thanks for the great idea Konstantin Boudnik!
          Building new images now.

          Show
          evans_ye Evans Ye added a comment - Committed and pushed. Thanks for the great idea Konstantin Boudnik ! Building new images now.

            People

            • Assignee:
              evans_ye Evans Ye
              Reporter:
              evans_ye Evans Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development