Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9107

Reduce time spent downloading maven artifacts for precommit tests

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 3.4.0
    • Impala 4.0.0
    • Frontend, Infrastructure
    • None
    • ghx-label-12

    Description

      When building Impala from scratch with an empty .m2 directory, maven needs to download a large number of jars and pom files. This is taking a long time and adds about 15-20 minutes to the build. For example, here is some output from a recent ubuntu-16.04-from-scratch, there is an 18 minute delay between the end of building the backend tests and the end of building the frontend:

      00:58:33 [100%] Built target hash-ring-util
      ...
      01:16:37 [100%] Built target fe

      Almost all of that time is spent downloading maven artifacts. Quite a few come from the maven central repository.

      This is taking way too much time, and we need to reduce it. The total size of artifacts being downloaded is not large. One approach would be to produce a tarball with the jars/poms that don't come from the CDP or CDH GBN repos. We can download that tarball and use it to either populate the .m2 directory or to stash it in IMPALA_TOOLCHAIN and use it as a maven repository with a file:// URI.

      This impacts all our jobs: ubuntu-16.04-from-scratch, ubuntu-16.04-dockerised-tests, all-build-options-ub1604, ubuntu-16.04-build-only, clang-tidy-ub1604

       

      Attachments

        Issue Links

          Activity

            People

              joemcdonnell Joe McDonnell
              joemcdonnell Joe McDonnell
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: