Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30970

Fix NPE in resolving k8s master url

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.4, 2.4.5, 3.0.0
    • Fix Version/s: 2.4.6, 3.0.0
    • Component/s: Kubernetes, Spark Core
    • Labels:
      None

      Description

      ```
      bin/spark-sql --master  k8s:///https://kubernetes.docker.internal:6443 --conf spark.kubernetes.container.image=yaooqinn/spark:v2.4.4
      Exception in thread "main" java.lang.NullPointerException
      	at org.apache.spark.util.Utils$.checkAndGetK8sMasterUrl(Utils.scala:2739)
      	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:261)
      	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
      	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
      	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
      	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
      	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      ```
      

      Althrough k8s:///https://kubernetes.docker.internal:6443 is a wrong master url but should not throw npe

        Attachments

          Activity

            People

            • Assignee:
              Qin Yao Kent Yao
              Reporter:
              Qin Yao Kent Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: