[YUNIKORN-726] Quick start results in admission controller crashloop back off.

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.2
• Labels: pull-request-available
• Assignee: Craig Condit
• Reporter: Holden Karau
• Created: 23/Jun/21 23:14
• Updated: 03/Feb/22 22:52
• Resolved: 21/Jan/22 21:40

Description

Following the quick start guide at https://yunikorn.apache.org/docs/ the admission controller fails start with an error indicating that it can't read the PEM data in the certificate input. Looking at the webhook-server-tls there is data in the key.pem but not cert.pem.

 

It looks like it does generate a certificate signing request:

 

(base) holden@hkdesktop:~/repos/incubator-yunikorn-k8shim$ kubectl describe certificatesigningrequest --all-namespaces
Name: yunikorn-admission-controller-service.yunikorn
Labels: <none>
Annotations: <none>
CreationTimestamp: Wed, 23 Jun 2021 16:03:43 -0700
Requesting User: system:serviceaccount:yunikorn:yunikorn-admin
Signer: kubernetes.io/legacy-unknown
Status: Pending
Subject:
 Common Name: yunikorn-admission-controller-service.yunikorn.svc
 Serial Number: 
Subject Alternative Names:
 DNS Names: yunikorn-admission-controller-service
 yunikorn-admission-controller-service.yunikorn
 yunikorn-admission-controller-service.yunikorn.svc
Events: <none>

 

This is on K3s 1.21

 

2021-06-23T23:09:33.107Z INFO log/logger.go:93 scheduler configuration, pretty print {"configs": "{\n \"clusterId\": \"my-kube-cluster\",\n \"clusterVersion\": \"0.1\",\n \"policyGroup\": \"queues\",\n \"schedulingIntervalSecond\": 1000000000,\n \"absoluteKubeConfigFilePath\": \"\",\n \"loggingLevel\": 0,\n \"logEncoding\": \"console\",\n \"logFilePath\": \"\",\n \"volumeBindTimeout\": 10000000000,\n \"testMode\": false,\n \"eventChannelCapacity\": 1048576,\n \"dispatchTimeout\": 300000000000,\n \"kubeQPS\": 1000,\n \"kubeBurst\": 1000,\n \"predicates\": \"\",\n \"operatorPlugins\": \"general,yunikorn-app\",\n \"enableConfigHotRefresh\": false\n}"}2021-06-23T23:09:33.107Z INFO log/logger.go:93 scheduler configuration, pretty print {"configs": "{\n \"clusterId\": \"my-kube-cluster\",\n \"clusterVersion\": \"0.1\",\n \"policyGroup\": \"queues\",\n \"schedulingIntervalSecond\": 1000000000,\n \"absoluteKubeConfigFilePath\": \"\",\n \"loggingLevel\": 0,\n \"logEncoding\": \"console\",\n \"logFilePath\": \"\",\n \"volumeBindTimeout\": 10000000000,\n \"testMode\": false,\n \"eventChannelCapacity\": 1048576,\n \"dispatchTimeout\": 300000000000,\n \"kubeQPS\": 1000,\n \"kubeBurst\": 1000,\n \"predicates\": \"\",\n \"operatorPlugins\": \"general,yunikorn-app\",\n \"enableConfigHotRefresh\": false\n}"}2021-06-23T23:09:33.107Z FATAL webhook/webhook.go:56 Failed to load key pair {"error": "tls: failed to find any PEM data in certificate input"}main.main /Users/boyuan/go/src/yunikorn-release-v0.10.0/incubator-yunikorn-release/staging/apache-yunikorn-0.10.0-incubating-src/k8shim/pkg/plugin/admissioncontrollers/webhook/webhook.go:56runtime.main /Users/boyuan/go/install/go1.14.14/src/runtime/proc.go:203

Issue Links

• is caused by: YUNIKORN-625 Use v1 for CertificateSigningRequest instead of v1beta1
• is related to: YUNIKORN-674 Admission controller could not be started on K8s 1.19
• is superceded by: YUNIKORN-941 split scheduler and admission controller deployment
• links to: GitHub Pull Request #59