[BEAM-10256] Experiment with Jenkins Worker VM Sizes and # executors

• Type: Improvement
• Status: In Progress
• Priority: P3
• Resolution: Unresolved
• Component/s: build-system
• Labels: Jenkins infra infrastructure jenkins
• Assignee: Unassigned
• Reporter: Tyson Hamilton
• Created: 12/Jun/20 22:10
• Updated: 19/Sep/24 10:01

Description

When on the new Apache infra for Jenkins we should revisit the size of our VMs. Currently we use n1-highmem-16 but this may be wasteful since only 2 executors are running on each worker.

 

One option is to increase the number of executors on each worker, say to 8, so each executor has 2 vcpus. The upper limit is our desired thread parallelism per executor (i.e. tests) and the memory required for each.

Another option is to downsize the VM to a n1-highmem-4 and keep 2 executors.

There are probably others, we should do some analysis to see what makes sense.