[FLINK-21942] KubernetesLeaderRetrievalDriver not closed after terminated which lead to connection leak

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.2, 1.13.0
• Fix Version/s: 1.13.0, 1.12.3
• Component/s: Runtime / Coordination
• Labels: k8s-ha pull-request-available
• Assignee: Yang Wang
• Reporter: Yi Tang
• Created: 24/Mar/21 03:57
• Updated: 28/Aug/21 11:13
• Resolved: 01/Apr/21 17:25

Description

Looks like KubernetesLeaderRetrievalDriver is not closed even if the KubernetesLeaderElectionDriver is closed and job reach globally terminated.
This will lead to many configmap watching be still active with connections to K8s.

When the connections exceeds max concurrent requests, those new configmap watching can not be started. Finally leads to all new jobs submitted timeout.

fly_in_gis trohrmann This may be related to FLINK-20695, could you confirm this issue?
But when many jobs are running in same session cluster, the config map watching is required to be active. Maybe we should merge all config maps watching?

Issue Links

• is related to: FLINK-22006 Could not run more than 20 jobs in a native K8s session when K8s HA enabled
• is related to: FLINK-22054 Using a shared watcher for ConfigMap watching
• relates to: FLINK-20695 Zookeeper node under leader and leaderlatch is not deleted after job finished
• links to: GitHub Pull Request #15407