[SPARK-22778] Kubernetes scheduler at master failing to run applications successfully

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: Kubernetes, Spark Core
• Assignee: Yinan Li
• Reporter: Anirudh Ramanathan
• Created: 13/Dec/17 20:14
• Updated: 17/May/20 18:23
• Resolved: 14/Dec/17 22:03

Description

Building images based on master and deploying Spark PI results in the following error.

2017-12-13 19:57:19 INFO SparkContext:54 - Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: Could not parse Master URL: 'k8s:https://xx.yy.zz.ww'
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2741)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:496)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2490)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:927)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:918)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:918)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
2017-12-13 19:57:19 INFO ShutdownHookManager:54 - Shutdown hook called
2017-12-13 19:57:19 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-b47515c2-6750-4a37-aa68-6ee12da5d2bd

This is likely an artifact seen because of changes in master, or our submission code in the reviews. We haven't seen this on our fork. Hopefully once integration tests are ported against upstream/master, we will catch these issues earlier.

Issue Links

• links to: [Github] Pull Request #19972 (liyinan926)