[IMPALA-13329] Add a mode to bin/single_node_perf_run.py to use cpusets to isolate impalads

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 4.5.0
• Component/s: Infrastructure
• Epic Color: ghx-label-8
• Assignee: Unassigned
• Reporter: Joe McDonnell
• Created: 26/Aug/24 17:33
• Updated: 26/Aug/24 17:57

Description

We want the perf-AB-test job to have as little noise and bias as possible to get a clean read on performance differences. One way to reduce the noise would be to use dedicated CPUs for each Impalad while they are running. Keeping the minicluster components and perf script runner separate avoids interactions between the two. Keeping the impalads separate from each other better simulates different physical machines with dedicated CPUs.

One way to do this is to use cpusets via cgroups. Essentially, we can create a separate cpuset cgroup for each impalad and another for everything else. For example, on a 16 CPU machine, one could give 4 CPUs to each of the three impalads and then have the remaining 4 CPUs available for everything else. This aligns with the hyperthreading boundaries, so nothing is sharing anything.