[IMPALA-6070] Speed up test execution

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Infrastructure
• Epic Color: ghx-label-2
• Assignee: Philip Martin
• Reporter: Philip Martin
• Created: 17/Oct/17 23:14
• Updated: 21/Dec/20 19:15

Description

Our tests (e.g., https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/buildTimeTrend) tend to take about 4 hours. This can be improved.

I'm opening this JIRA track those changes. I'm currently looking at:

• Parallelizing multiple data-load steps: TPC-DS, TPC-H, and Functional take ~65 minutes when serialized. They take 35 minutes if running in parallel.
• Parallelizing compute stats: this takes ~10 minutes; probably can be faster.

The trickier thing is parallelizing fe tests, ee tests, and custom cluster tests. The approach I'm taking is to create a docker container with everything in it (including data load), and then running tests in parallel. This is a bit messier, but I think it has some legs when it comes to using machines with many cores.