[SPARK-11201] StreamContext.getOrCreate is broken is yarn-client mode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.5.1
• Component/s: DStreams
• Target Version/s: 1.5.2
• Assignee: Unassigned
• Reporter: Hari Shreedharan
• Created: 20/Oct/15 05:32
• Updated: 20/Oct/15 18:50
• Resolved: 20/Oct/15 18:50

Description

If StreamingContext.getOrCreate (or the constructors that create the Hadoop Configuration is used, SparkHadoopUtil.get.conf is called before SparkContext is created - when SPARK_YARN_MODE is set. So in that case SparkHadoopUtil.get creates a SparkHadoopUtil instance instead of YarnSparkHadoopUtil instance.

So, in yarn-client mode, a class cast exception gets thrown from Client.scala:

java.lang.ClassCastException: org.apache.spark.deploy.SparkHadoopUtil cannot be cast to org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
	at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:169)
	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:266)
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:631)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:120)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:523)
	at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:854)
	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
	at com.cloudera.test.LongRunningApp$.com$cloudera$test$LongRunningApp$$createCheckpoint$1(LongRunningApp.scala:33)
	at com.cloudera.test.LongRunningApp$$anonfun$4.apply(LongRunningApp.scala:90)
	at com.cloudera.test.LongRunningApp$$anonfun$4.apply(LongRunningApp.scala:90)
	at scala.Option.getOrElse(Option.scala:120)
	at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:844)
	at com.cloudera.test.LongRunningApp$.main(LongRunningApp.scala:90)
	at com.cloudera.test.LongRunningApp.main(LongRunningApp.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Issue Links

• duplicates: SPARK-10812 Spark Hadoop Util does not support stopping a non-yarn Spark Context & starting a Yarn spark context.
• links to: [Github] Pull Request #9174 (harishreedharan)