[FLINK-22054] Using a shared watcher for ConfigMap watching

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.1, 1.12.4
• Fix Version/s: 1.14.0
• Component/s: Deployment / Kubernetes, Runtime / Coordination
• Labels: k8s-ha pull-request-available
• Assignee: Yi Tang
• Reporter: Yi Tang
• Created: 31/Mar/21 02:59
• Updated: 05/May/23 10:47
• Resolved: 23/Jul/21 02:07

Description

While using K8s HA service, the watching for ConfigMap is separate for each job. As the number of running jobs increases, this consumes a large amount of connections.

Here we proposal to use a shared watcher for each FlinkKubeClient, and dispatch events to different listeners. At the same time, we should keep the same semantic with watching separately.

Issue Links

• causes: FLINK-24819 Higher APIServer cpu load after using SharedIndexInformer replaced naked Kubernetes watch
• is related to: FLINK-32010 KubernetesLeaderRetrievalDriver always waits for lease update to resolve leadership
• relates to: FLINK-22006 Could not run more than 20 jobs in a native K8s session when K8s HA enabled
• relates to: FLINK-21942 KubernetesLeaderRetrievalDriver not closed after terminated which lead to connection leak
• links to: GitHub Pull Request #15501
• links to: GitHub Pull Request #16661