Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
Description
The YuniKorn Helm chart can be successfully installed with helm install, but if helm upgrade is used to upgrade an existing installation, then the admission controller pod will be gone.
For example, if an existing installation looks like below:
$ kubectl get pods NAME READY STATUS RESTARTS AGE yunikorn-admission-controller-6f6b9b8dff-thv8k 1/1 Running 0 31s yunikorn-scheduler-99969db67-bxhmf 2/2 Running 0 34s
Then after some changes are made to the Helm chart (e.g. changing queue configurations) and helm upgrade is executed to make the changes take effect, the end result will be:
$ kubectl get pods NAME READY STATUS RESTARTS AGE yunikorn-scheduler-54dd4b45d7-b9jg4 2/2 Running 0 24s
Note that the scheduler pod has been correctly replaced with a new one, but the admission controller pod is gone (the old admission controller pod has been deleted as expected, but no new pod was spawned).
Looks like the reason it didn't work was these lifecycle hooks: https://github.com/apache/incubator-yunikorn-release/blob/v0.10.0/helm-charts/yunikorn/templates/deployment.yaml#L83-L89. I think helm upgrade first starts a new scheduler pod, runs the postStart hook to create a new admission controller. Then helm deletes the old scheduler pod. After it's gone, helm runs the postStop hook of the old scheduler pod, which caused the unintentional side effect of deleting both the old and new admission controllers, leaving no admission controller around.
It would be great if this issue can be fixed because helm upgrade is the standard way of upgrading a helm release. It can also completely replace helm install with helm upgrade --install when it comes to a fresh installation
Attachments
Issue Links
- links to