[SPARK-34264] Prevent incomplete master URLs for Spark on Kubernetes early

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.0.1, 3.1.1
• Component/s: Kubernetes, Spark Submit
• Assignee: Unassigned
• Reporter: Jacek Laskowski
• Created: 27/Jan/21 13:40
• Updated: 27/Jan/21 13:40

Description

It turns out that --master k8s:// is accepted and although leads to termination displays stacktraces that don't really tell what the real cause is.

This may happen when the Kubernetes API server(s) are described by an environment variable that's not initialized in the current terminal.

$ ./bin/spark-shell --master k8s:// --verbose
...
Spark config:
(spark.jars,)
(spark.app.name,Spark shell)
(spark.submit.pyFiles,)
(spark.ui.showConsoleProgress,true)
(spark.submit.deployMode,client)
(spark.master,k8s://https://)
...
21/01/27 14:29:44 ERROR Main: Failed to initialize Spark session.
io.fabric8.kubernetes.client.KubernetesClientException: Failed to start websocket
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:208)
...
Caused by: java.net.UnknownHostException: api: nodename nor servname provided, or not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
	at okhttp3.Dns$1.lookup(Dns.java:40)
	at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)