[FLINK-28124] Flink on K8S in hosetNetwork mode, the CPU usage will spike

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Deployment / Kubernetes, Runtime / Metrics
• Assignee: Unassigned
• Reporter: liuzhuo
• Created: 20/Jun/22 03:22
• Updated: 20/Jun/22 03:22

Description

When using the flink on k8s, in the configuration 'kubernetes. Hostnetwork. Enabled = true' parameters, If a node is running too many pods
, there will be a node CPU surge problem, The metric module collected the indicator information of the network card

 

Class :SystemResourcesCounter
Line 109 :calculateNetworkUsage(hardwareAbstractionLayer.getNetworkIFs());

The problem is with this method

hardwareAbstractionLayer.getNetworkIFs()

this method returns the current all the  network card on the machine, each pod will create a virtual network card (ifr_name = "kube - ipvs0"), In hostNetwork mode, the network card is visible to the public, resulting in too much network card information returned by the interface

For example, if a node runs 200 pods, then the method calling this interface in each POD will return at least 200 nic information. With the default execution once every 5 seconds, this machine needs to return 200*200=40000 network card information every 5 seconds, this results in a significant CPU consumption