Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31482

spark.kubernetes.driver.podTemplateFile Configuration not used by the job

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Kubernetes, Spark Core
    • Labels:
      None

      Description

      Spark 3.0 - Running Spark Submit as below and point to a MinKube cluster

      bin/spark-submit \
       --master k8s://https://192.168.99.102:8443 \
       --deploy-mode cluster \
       --name spark-pi \
       --class org.apache.spark.examples.SparkPi \
       --conf spark.kubernetes.driver.podTemplateFile=../driver_1E.template \
       --conf spark.kubernetes.executor.podTemplateFile=../executor.template \
       --conf spark.kubernetes.container.image=spark:spark3 \
       local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-preview2.jar 1
      

       

      Spark Binaries - spark-3.0.0-preview2-bin-hadoop2.7.tgz

      Driver Template - 

      apiVersion: v1
      kind: Pod
      metadata:
        labels:
          spark-app-id: my-custom-id
        annotations:
          spark-driver-cpu: 1
          spark-driver-mem: 1
          spark-executor-cpu: 1
          spark-executor-mem: 1
          spark-executor-count: 1
      spec:
        schedulerName: spark-scheduler

       Executor Template

       

      apiVersion: v1
      kind: Pod
      metadata:
        labels:
          spark-app-id: my-custom-id
      spec:
        schedulerName: spark-scheduler

      Kubernetes Pods Launched - Two Executor Pods were launched which was default

      spark-pi-e608e7718f11cc69-driver   1/1     Running     0          10s
      spark-pi-e608e7718f11cc69-exec-1   1/1     Running     0          5s
      spark-pi-e608e7718f11cc69-exec-2   1/1     Running     0          5s

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              pradeep.misra@mail.com Pradeep Misra
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: