[SPARK-39023] Add Executor Pod inter-pod anti-affinity

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.2.1
• Component/s: Kubernetes
• Assignee: Unassigned
• Reporter: binjie yang
• Created: 26/Apr/22 09:13
• Updated: 20/May/22 03:46
• Resolved: 20/May/22 03:46

Description

Why should we need this?

When Spark On Kubernetes is running, Executor Pod clusters occur in certain conditions (uneven resource allocation in Kubernetes, high load On some nodes, low load On some nodes), causing Shuffle data skew. This causes Spark application to fail or performance bottlenecks, such as Shuffle Fetch timeout and connection refuse after connection number.

Why should use this?

The functionality mentioned in this PR was tested on a cluster.

Using three Kubernetes Node(node-1, node-2, node-3).

In the case of sufficient or insufficient cluster resources, it  has the same effect Whether the feature is enabled or not. Kubernetes assigns pods to nodes with low load based on global resources. When only one node has a small load, Kubernetes will schedule all executor pods to this node. 

Issue Links

• duplicates: SPARK-24434 Support user-specified driver and executor pod templates
• links to: [Github] Pull Request #36358 (zwangsheng)
• links to: [Github] Pull Request #36358 (zwangsheng)