[SAMZA-2779] Container CPU metrics inconsistent.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Adam Faris
• Created: 19/Apr/23 00:22
• Updated: 19/Apr/23 02:49

Description

The updateOperatingSystemMetrics method is intended to report CPU usage for the container.
https://github.com/apache/samza/blob/e37fe259cede587928cc0d543fc8384250c7e113/samza-core/src/main/scala/org/apache/samza/metrics/JvmMetrics.scala#L169

gProcessCpuUsageProcessors.set(processCpuLoad * operatingSystemMXBean.getAvailableProcessors) 

gSystemCpuUsage.set(operatingSystemMXBean.getSystemCpuLoad * PCT) 

• Java docs for operatingSystemMXBean.getAvailableProcessors

returns the number of processors available to the virtual machine

So what does this mean to Samza Metrics?

gProcessCpuUsageProcessors.set(processCpuLoad * operatingSystemMXBean.getAvailableProcessors)

The multiplier will change based on the number of processors a host will have. Some hosts may have 24 cores, others 40. As the container moves form hostA to hostB, this value will lose meaning. 

Secondly it's not clear what ProcessCpuUsageProcessors represents.  Why would it be multiplied by the CPU count?

Here's a simple example class to demonstrate the behavior.

import java.lang.management.ManagementFactory;
import com.sun.management.OperatingSystemMXBean;

public class Count {

    public static void main(String[] args) {
        OperatingSystemMXBean osMXBean = (OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
        int processorCount = osMXBean.getAvailableProcessors();
        System.out.printf("Processor Count: %d\n", processorCount);
    }

}