[SPARK-39457] Support IPv6-only environment

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Labels: releasenotes
• Assignee: Dongjoon Hyun
• Reporter: DB Tsai
• Created: 13/Jun/22 18:24
• Updated: 27/Jul/22 01:46
• Resolved: 27/Jul/22 01:41

Description

Spark doesn't fully work in pure IPV6 environment that doesn't have IPV4 at all. This is an umbrella jira tracking the support of pure IPV6 deployment.

 

The scope of this JIRA doesn't include ancient resource managers like Mesos(deprecated in Spark) or YARN(because Apache Spark still supports Hadoop 2). We are going to focus on only `Spark Standalone` and `K8s`

 

K8s IPv4/IPv6 dual-stack Feature reached `Stable` stage at v1.23.

• https://kubernetes.io/docs/concepts/services-networking/dual-stack/
  • v1.16 [alpha]
  • v1.21 [beta]
  • v1.23 [stable]

 

Currently, EKS supports IPv6 with v1.21.13-eks-84b4fe6.

• https://aws.amazon.com/blogs/containers/amazon-eks-launches-ipv6-support/