Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34264

Prevent incomplete master URLs for Spark on Kubernetes early

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.1, 3.1.1
    • Fix Version/s: None
    • Component/s: Kubernetes, Spark Submit
    • Labels:
      None

      Description

      It turns out that --master k8s:// is accepted and although leads to termination displays stacktraces that don't really tell what the real cause is.

      This may happen when the Kubernetes API server(s) are described by an environment variable that's not initialized in the current terminal.

      $ ./bin/spark-shell --master k8s:// --verbose
      ...
      Spark config:
      (spark.jars,)
      (spark.app.name,Spark shell)
      (spark.submit.pyFiles,)
      (spark.ui.showConsoleProgress,true)
      (spark.submit.deployMode,client)
      (spark.master,k8s://https://)
      ...
      21/01/27 14:29:44 ERROR Main: Failed to initialize Spark session.
      io.fabric8.kubernetes.client.KubernetesClientException: Failed to start websocket
      	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:208)
      ...
      Caused by: java.net.UnknownHostException: api: nodename nor servname provided, or not known
      	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
      	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
      	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
      	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
      	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
      	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
      	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
      	at okhttp3.Dns$1.lookup(Dns.java:40)
      	at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)
      

        Attachments

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              Unassigned Assign to me
              Reporter:
              jlaskowski Jacek Laskowski

              Dates

              • Created:
                Updated:

                Issue deployment