[FLINK-24819] Higher APIServer cpu load after using SharedIndexInformer replaced naked Kubernetes watch

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.19.0, 1.18.1, 1.17.3
• Component/s: Deployment / Kubernetes
• Assignee: Unassigned
• Reporter: Yang Wang
• Created: 08/Nov/21 09:44
• Updated: 06/Dec/23 03:12
• Resolved: 06/Dec/23 03:12

Description

In FLINK-22054, Flink has used a shared informer for ConfigMap to replace the naked K8s watch. After then, each Flink JVM process(JM/TM) only needs one connection to APIServer for ConfigMap watching. It aims to reduce the network pressure on K8s APIServer.

 

However, in our recent tests, we found that the CPU and memory cost of APIServer have been doubled while running same Flink workloads. After digging more details in the K8s, I think the root cause might be that ETCD does not have indexes for labels. It means APIServer need to pull all the events from ETCD for each watch and then filter with specified labels(e.g. app=xxx,type=flink-native-kubernetes,configmap-type=high-availability) internally. Before FLINK-22054, we started a dedicated connection for each ConfigMap watching. And it seems that APIServer only need to pull the events for the specified ConfigMap name.

 

Watch URL example(Before):

https://kubernetes.default:6443/api/v1/namespaces/vvp-workload/configmaps?metadata.name=job-009d4f51-ca02-4793-a49b-a3344538719b-resourcemanager-leader&watch=true

 

Watch URL example(After):

https://kubernetes.default:6443/api/v1/namespaces/vvp-workload/configmaps?labelSelector=app%3Dk8s-ha-app-1-1636077491-23461%2Ctype%3Dflink-native-kubernetes%2Cconfigmap-type%3Dhigh-availability&resourceVersion=1153687321&watch=true

Issue Links

• is caused by: FLINK-22054 Using a shared watcher for ConfigMap watching
• is fixed by: FLINK-33598 Watch HA configmap via name instead of lables to reduce pressure on APIserver