[IMPALA-9107] Reduce time spent downloading maven artifacts for precommit tests

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 4.0.0
• Component/s: Frontend, Infrastructure
• Epic Color: ghx-label-12
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 30/Oct/19 19:44
• Updated: 16/Jul/22 04:22
• Resolved: 05/Oct/20 18:38

Description

When building Impala from scratch with an empty .m2 directory, maven needs to download a large number of jars and pom files. This is taking a long time and adds about 15-20 minutes to the build. For example, here is some output from a recent ubuntu-16.04-from-scratch, there is an 18 minute delay between the end of building the backend tests and the end of building the frontend:

00:58:33 [100%] Built target hash-ring-util
...
01:16:37 [100%] Built target fe

Almost all of that time is spent downloading maven artifacts. Quite a few come from the maven central repository.

This is taking way too much time, and we need to reduce it. The total size of artifacts being downloaded is not large. One approach would be to produce a tarball with the jars/poms that don't come from the CDP or CDH GBN repos. We can download that tarball and use it to either populate the .m2 directory or to stash it in IMPALA_TOOLCHAIN and use it as a maven repository with a file:// URI.

This impacts all our jobs: ubuntu-16.04-from-scratch, ubuntu-16.04-dockerised-tests, all-build-options-ub1604, ubuntu-16.04-build-only, clang-tidy-ub1604

 

Issue Links

• is related to: IMPALA-10668 Provide scripts for offline compiling
• relates to: IMPALA-9558 Add m2 mountpoint to the Docker-based test infra
• relates to: IMPALA-11439 Add an environment variable to control bootstrap_system.sh whether to prepopulate the m2 directory or not