[FLINK-22006] Could not run more than 20 jobs in a native K8s session when K8s HA enabled

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.12.2, 1.13.0
• Fix Version/s: 1.13.0, 1.12.3
• Component/s: Runtime / Coordination
• Labels: k8s-ha pull-request-available
• Assignee: Yang Wang
• Reporter: Yang Wang
• Created: 29/Mar/21 06:28
• Updated: 28/May/21 09:00
• Resolved: 07/Apr/21 16:45

Description

Currently, if we start a native K8s session cluster when K8s HA enabled, we could not run more than 20 streaming jobs. 

 

The latest job is always initializing, and the previous one is created and waiting to be assigned. It seems that some internal resources have been exhausted, e.g. okhttp thread pool , tcp connections or something else.

Issue Links

• causes: FLINK-22047 Could not find FLINSHED Flink job and can't submit job
• is related to: FLINK-22054 Using a shared watcher for ConfigMap watching
• relates to: FLINK-21942 KubernetesLeaderRetrievalDriver not closed after terminated which lead to connection leak
• links to: GitHub Pull Request #15480