Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11201

StreamContext.getOrCreate is broken is yarn-client mode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.5.1
    • Fix Version/s: None
    • Component/s: DStreams
    • Labels:
      None
    • Target Version/s:

      Description

      If StreamingContext.getOrCreate (or the constructors that create the Hadoop Configuration is used, SparkHadoopUtil.get.conf is called before SparkContext is created - when SPARK_YARN_MODE is set. So in that case SparkHadoopUtil.get creates a SparkHadoopUtil instance instead of YarnSparkHadoopUtil instance.

      So, in yarn-client mode, a class cast exception gets thrown from Client.scala:

      java.lang.ClassCastException: org.apache.spark.deploy.SparkHadoopUtil cannot be cast to org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
      	at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:169)
      	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:266)
      	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:631)
      	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:120)
      	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
      	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
      	at org.apache.spark.SparkContext.<init>(SparkContext.scala:523)
      	at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:854)
      	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
      	at com.cloudera.test.LongRunningApp$.com$cloudera$test$LongRunningApp$$createCheckpoint$1(LongRunningApp.scala:33)
      	at com.cloudera.test.LongRunningApp$$anonfun$4.apply(LongRunningApp.scala:90)
      	at com.cloudera.test.LongRunningApp$$anonfun$4.apply(LongRunningApp.scala:90)
      	at scala.Option.getOrElse(Option.scala:120)
      	at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:844)
      	at com.cloudera.test.LongRunningApp$.main(LongRunningApp.scala:90)
      	at com.cloudera.test.LongRunningApp.main(LongRunningApp.scala)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
      	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
      	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hshreedharan Hari Shreedharan
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: