[BEAM-10256] Experiment with Jenkins Worker VM Sizes and # executors

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: build-system
• Labels: Jenkins infra infrastructure jenkins
• Assignee: Unassigned
• Reporter: Tyson Hamilton
• Created: 12/Jun/20 22:10
• Updated: 04/Jun/22 16:37

Description

When on the new Apache infra for Jenkins we should revisit the size of our VMs. Currently we use n1-highmem-16 but this may be wasteful since only 2 executors are running on each worker.

 

One option is to increase the number of executors on each worker, say to 8, so each executor has 2 vcpus. The upper limit is our desired thread parallelism per executor (i.e. tests) and the memory required for each.

Another option is to downsize the VM to a n1-highmem-4 and keep 2 executors.

There are probably others, we should do some analysis to see what makes sense.