[SPARK-24353] Add support for pod affinity/anti-affinity

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.3.0
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Stavros Kontopoulos
• Created: 22/May/18 16:26
• Updated: 17/May/20 18:24
• Resolved: 15/Feb/19 01:00

Description

Spark on K8s allows to place driver/executor pods on specific k8s nodes, using nodeSelector. NodeSelector is a very simple way to constrain pods to nodes with particular labels. The affinity/anti-affinity feature, currently in beta, greatly expands the types of constraints you can express. Aim here is to bring support of this feature to Spark on K8s, in detail:

• Node affinity
• Toleration/taints
• Inter-Pod affinity/anti-affinity

Note that nodeSelector will be deprecated in the future.

Design doc: https://docs.google.com/document/d/1izk75I4A0I-nJaE57m7wkpgUZTXM0o6-c0Tb-NxdOMU

 

 

Issue Links

• links to: GitHub Pull Request #26505