[FLINK-33880] Introducing Retry Mechanism for Listing TaskManager Pods to Prevent API Server Connection Failures

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.17.2
• Component/s: Deployment / Kubernetes
• Assignee: Yuan Huang
• Reporter: Yuan Huang
• Created: 19/Dec/23 10:45
• Updated: 02/Jan/24 03:39

Description

When operating in Kubernetes mode, if the JobManager undergoes a restart, it attempts to establish a connection with the API server to retrieve the complete list of TaskManager Pods, facilitating the recovery of previous TaskManagers.

In the context of a large Kubernetes cluster with potentially thousands of concurrently running jobs, a scenario may arise where all JobManagers undergo a restart and subsequently connect to the API server (e.g., during disaster recovery). This influx of requests may overwhelm the API server, reaching its maximum capacity and leading to the refusal of some JobManager requests. Consequently, certain JobManagers may experience failures and initiate reconnection attempts to the API server.

To enhance this process, we can propose the implementation of a retry mechanism. In the event of a failed connection attempt to the API server, Flink will introduce a waiting period before making subsequent connection attempts, mitigating the risk of overwhelming the server and improving the overall resilience of the system.