[SPARK-24905] Spark 2.3 Internal URL env variable

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.1
• Component/s: Kubernetes, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Björn Wenzel
• Created: 24/Jul/18 12:12
• Updated: 12/Dec/22 18:10
• Resolved: 08/Oct/19 05:42

Description

Currently the Kubernetes Master internal URL is hardcoded in the Constants.scala file (https://github.com/apache/spark/blob/85fe1297e35bcff9cf86bd53fee615e140ee5bfb/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L75)

In some cases these URL should be changed e.g. if the Certificate is valid for another Hostname.

Is it possible to make this URL a property like: spark.kubernetes.authenticate.driver.hostname?

Kubernetes The Hard Way maintained by Kelsey Hightower for example uses kubernetes.default as hostname, this will produce again a SSLPeerUnverifiedException.

 

Here is the use of the Hardcoded Host: https://github.com/apache/spark/blob/85fe1297e35bcff9cf86bd53fee615e140ee5bfb/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala#L52 maybe this could be changed like the KUBERNETES_NAMESPACE property in Line 53.