Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35828

Should skip retrieving driver pod in client mode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.2, 3.1.2, 3.2.0
    • Fix Version/s: None
    • Component/s: Kubernetes
    • Labels:
      None

      Description

      bin/spark-submit  \
      --conf spark.kubernetes.file.upload.path=./ \
      --deploy-mode client \
      --master k8s://https://kubernetes.docker.internal:6443 \
      --conf spark.kubernetes.container.image=yaooqinn/spark:v20210619 \
      -c spark.kubernetes.context=docker-for-desktop_1 \
      --conf spark.kubernetes.executor.podNamePrefix=sparksql \
      --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
      --conf spark.dynamicAllocation.enabled=true \
      --conf spark.kubernetes.driver.pod.name=abc \
      --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.12-3.2.0-SNAPSHOT.jar
      

      When `spark.kubernetes.driver.pod.name` is specific, we now get the driver pod for whatever the deploy mode is, while the driver pod only exists in cluster mode. So we should skip retrieving it instead of get the following error.

      21/06/19 16:18:49 ERROR SparkContext: Error initializing SparkContext.
      org.apache.spark.SparkException: No pod was found named abc in the cluster in the namespace default (this was supposed to be the driver pod.).
      	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$driverPod$2(ExecutorPodsAllocator.scala:81)
      	at scala.Option.getOrElse(Option.scala:189)
      	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$driverPod$1(ExecutorPodsAllocator.scala:79)
      	at scala.Option.map(Option.scala:230)
      	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.<init>(ExecutorPodsAllocator.scala:76)
      	at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:118)
      	at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2969)
      	at org.apache.spark.SparkContext.<init>(SparkContext.scala:559)
      	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2686)
      	at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:948)
      	at scala.Option.getOrElse(Option.scala:189)
      	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:942)
      	at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)
      	at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
      	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
      	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
      	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
      	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
      	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Qin Yao Kent Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: