Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31800

Unable to disable Kerberos when submitting jobs to Kubernetes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Kubernetes
    • Labels:
      None

      Description

      When you attempt to submit a process to Kubernetes using spark-submit through --master, it returns the exception:

      20/05/22 20:25:54 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
      Exception in thread "main" org.apache.spark.SparkException: Please specify spark.kubernetes.file.upload.path property.
              at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadFileUri(KubernetesUtils.scala:290)
              at org.apache.spark.deploy.k8s.KubernetesUtils$.$anonfun$uploadAndTransformFileUris$1(KubernetesUtils.scala:246)
              at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
              at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
              at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
              at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
              at scala.collection.TraversableLike.map(TraversableLike.scala:238)
              at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
              at scala.collection.AbstractTraversable.map(Traversable.scala:108)
              at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadAndTransformFileUris(KubernetesUtils.scala:245)
              at org.apache.spark.deploy.k8s.features.BasicDriverFeatureStep.$anonfun$getAdditionalPodSystemProperties$1(BasicDriverFeatureStep.scala:165)
              at scala.collection.immutable.List.foreach(List.scala:392)
              at org.apache.spark.deploy.k8s.features.BasicDriverFeatureStep.getAdditionalPodSystemProperties(BasicDriverFeatureStep.scala:163)
              at org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:60)
              at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
              at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
              at scala.collection.immutable.List.foldLeft(List.scala:89)
              at org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.buildFromFeatures(KubernetesDriverBuilder.scala:58)
              at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:98)
              at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4(KubernetesClientApplication.scala:221)
              at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4$adapted(KubernetesClientApplication.scala:215)
              at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2539)
              at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:215)
              at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:188)
              at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
              at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
              at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
              at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
              at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
              at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
              at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      20/05/22 20:25:54 INFO ShutdownHookManager: Shutdown hook called
      20/05/22 20:25:54 INFO ShutdownHookManager: Deleting directory /private/var/folders/p1/y24myg413wx1l1l52bsdn2hr0000gq/T/spark-c94db9c5-b8a8-414d-b01d-f6369d31c9b8 

      No changes in settings appear to be able to disable Kerberos. This is when running a simple execution of the SparkPi on our lab cluster. The command being used is

      ./bin/spark-submit --master k8s://https://{api_hostname} --deploy-mode cluster --name spark-test --class org.apache.spark.examples.SparkPi --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.namespace=spark-jobs --conf spark.executor.instances=5 --conf spark.kubernetes.container.image={docker_registry}/spark:spark-3-test /opt/spark/examples/jars/spark-examples_2.12-3.0.0-preview2.jar

      It is important to note that this same command, when run on Spark 2.4.5 works flawlessly once all of the RBAC accounts are properly setup in Kubernetes.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              drahkar James Boylan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: