[SPARK-38668] Spark on Kubernetes: add separate pod watcher service to reduce pressure on K8s API server

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.1
• Component/s: Kubernetes
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: BoYang
• Created: 27/Mar/22 19:50
• Updated: 13/Dec/23 15:47

Description

Spark driver will listen to all pods events to manage its executor pods. This will cause pressure on Kubernetes API server in a large cluster, because there will be many drivers connect to the API server and watch for the pods.

 

An alternative is to have a separate service to listen and watch all pod events. Then each Spark driver only connects to that service to get pod events. This will reduce the load on Kubernetes API server.

Issue Links

• links to: GitHub Pull Request #44304