Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.4.4
-
None
-
A kuberentes cluster that has been in use for over 2 years and handles large amounts of production payloads.
Description
spark submit can not be used to to schedule to kuberentes with oauth token and cacert
spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com \
--conf spark.kubernetes.authenticate.submission.oauthToken=$TOKEN \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.authenticate.submission.caCertFile=/home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt \
--conf spark.kubernetes.namespace=here-olp-3dds-sit \
--conf spark.executor.instances=1 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
returns
log4j:WARN No appenders could be found for logger (io.fabric8.kubernetes.client.Config). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:183) at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235) at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2542) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:104) at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:197) at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128) at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122) at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:78) ... 13 more Caused by: java.io.IOException: Empty input at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) ... 19 more
The cacert and token are both valid and work even with curl
curl --cacert /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt -H "Authorization: bearer $TOKEN" -v https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com/api/v1/namespaces/here-olp-3dds-sit/pods -o out % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.117.233.37:443... * TCP_NODELAY set * Connected to api.borg-dev-1-aws-eu-west-1.k8s.in.here.com (10.117.233.37) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [58 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [1565 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [556 bytes data] * TLSv1.2 (IN), TLS handshake, Request CERT (13): { [588 bytes data] * TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data] * TLSv1.2 (OUT), TLS handshake, Certificate (11): } [7 bytes data] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [37 bytes data] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data] * TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=kubernetes-master * start date: Apr 2 13:30:06 2019 GMT * expire date: Apr 1 13:30:06 2020 GMT * subjectAltName: host "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com" matched cert's "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com" * issuer: C=DE; ST=Berlin; O=HERE Global BV; OU=OLP Engineering Infrastructure Development; CN=borg-dev-1-aws-eu-west-1; emailAddress=sdp_ops_team@here.com * SSL certificate verify ok. 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 } [5 bytes data] * Using Stream ID: 1 (easy handle 0x558a2ede87b0) } [5 bytes data] > GET /api/v1/namespaces/here-olp-3dds-sit/pods HTTP/2 > Host: api.borg-dev-1-aws-eu-west-1.k8s.in.here.com > User-Agent: curl/7.66.0 > Accept: */* > Authorization: bearer ***************************************** > { [5 bytes data] * Connection state changed (MAX_CONCURRENT_STREAMS == 250)! } [5 bytes data] < HTTP/2 200 < audit-id: a26869e9-7b42-4013-b1fa-839e963c6b78 < content-type: application/json < date: Wed, 13 Nov 2019 20:20:18 GMT < { [5 bytes data] 100 56466 0 56466 0 0 64020 0 --:--:-- --:--:-- --:--:-- 63947 * Connection #0 to host api.borg-dev-1-aws-eu-west-1.k8s.in.here.com left intact
// code placeholder