[SPARK-39457] Support IPv6-only environment

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Labels: releasenotes
• Assignee: Dongjoon Hyun
• Reporter: DB Tsai
• Created: 13/Jun/22 18:24
• Updated: 27/Jul/22 01:46
• Resolved: 27/Jul/22 01:41

Description

Spark doesn't fully work in pure IPV6 environment that doesn't have IPV4 at all. This is an umbrella jira tracking the support of pure IPV6 deployment.

 

The scope of this JIRA doesn't include ancient resource managers like Mesos(deprecated in Spark) or YARN(because Apache Spark still supports Hadoop 2). We are going to focus on only `Spark Standalone` and `K8s`

 

K8s IPv4/IPv6 dual-stack Feature reached `Stable` stage at v1.23.

• https://kubernetes.io/docs/concepts/services-networking/dual-stack/
  • v1.16 [alpha]
  • v1.21 [beta]
  • v1.23 [stable]

 

Currently, EKS supports IPv6 with v1.21.13-eks-84b4fe6.

• https://aws.amazon.com/blogs/containers/amazon-eks-launches-ipv6-support/

Sub-Tasks

• local*HostName* methods should support IPv6: Resolved
• Improve RpcAddress to add [] to IPv6 if needed: Resolved
• SocketAuthServer should respect Java IPv6 options: Resolved
• Improve SparkContext to propagate `java.net.preferIPv6Addresses`: Resolved
• Support `ipFamilyPolicy` and `ipFamilies` in Driver Service: Resolved
• KubernetesExecutorBackend should handle IPv6 hostname: Resolved
• Add net-tools to Spark docker files: Resolved
• Support IPv6 between JVM and Python Daemon in PySpark: Resolved
• Improve YARN client mode to support IPv6: Resolved
• Improve YARN cluster mode to support IPv6: Resolved
• Support IPv6 in WebUI: Resolved
• Add build and test documentation on IPv6: Resolved
• Fix UISuite for IPv6: Resolved
• Fix CoarseGrainedSchedulerBackendSuite to handle fast allocations: Resolved
• Print `SPARK_LOCAL_(HOSTNAME|IP)` in `build/{mvn|sbt}`: Resolved
• Use `Utils.localCanonicalHostName` instead of `localhost` in tests: Resolved
• Ivy doesn't work correctly on IPv6-only environment: Closed
• Propagate `java.net.preferIPv6Addresses=true` in SBT tests: Resolved
• Support DEFAULT_ARTIFACT_REPOSITORY in check-license: Resolved
• LauncherBackendSuite should add java.net.preferIPv6Addresses conf: Resolved
• Fix KafkaTestUtils to support IPv6: Resolved
• Make hive-thrift server module passes in IPv6 environment: Resolved
• Use localHostNameForURI in UISuite: Resolved
• Fix `test_daemon.py` to support IPv6: Resolved
• Improve logs to show addresses in addition to port: Resolved
• Principals in KafkaTestUtils should use canonical host name: Resolved
• Add `IPv4 and IPv6` section to K8s document: Resolved
• IPv6 address mapping can fail with sparse addresses: Resolved