[FLINK-30881] Crictl/Minikube version mismatch causes errors in k8s setup

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.17.0, 1.15.3, 1.16.1
• Fix Version/s: 1.17.0, 1.15.4, 1.16.2
• Component/s: Deployment / Kubernetes, Test Infrastructure
• Labels: pull-request-available test-stability
• Assignee: Matthias Pohl
• Reporter: Matthias Pohl
• Created: 02/Feb/23 11:12
• Updated: 03/Feb/23 08:55
• Resolved: 03/Feb/23 07:45

Description

We observed constant failures in the e2e k8s tests with permission issues. This was initially accidentally reported through FLINK-29671. But FLINK-29671 actually covers a different instability.

Here are the build failures initially reported in FLINK-29671:

• https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45548&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14&l=4972
• https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45588&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14&l=4900
• https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45588&view=logs&j=af885ea8-6b05-5dc2-4a37-eab9c0d1ab09&t=f779a55a-0ffe-5bbc-8824-3a79333d4559&l=5597
• https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45587&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14&l=4818
• https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45587&view=logs&j=e9d3d34f-3d15-59f4-0e3e-35067d100dfe&t=f8a6d3eb-38cf-5cca-9a99-d0badeb5fe62&l=5852
• https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45591&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14&l=4915
• https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45598&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14&l=4921
• https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45603&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14&l=4991

Feb 01 11:00:45 Starting minikube ...
Feb 01 11:00:45 * minikube v1.29.0 on Ubuntu 20.04
Feb 01 11:00:45 * Using the none driver based on existing profile
Feb 01 11:00:45 * Starting control plane node minikube in cluster minikube
Feb 01 11:00:45 * Restarting existing none bare metal machine for "minikube" ...
Feb 01 11:00:45 * OS release is Ubuntu 20.04.5 LTS
Feb 01 11:01:22 
X Exiting due to RUNTIME_ENABLE: Temporary Error: sudo /usr/local/bin/crictl version: exit status 1
stdout:
[...]  
Feb 01 11:01:22 
E0201 11:01:22.809164  241870 root.go:80] failed to log command start to audit: failed to open the audit log: open /home/vsts/.minikube/logs/audit.json: permission denied
Feb 01 11:01:22 
X Exiting due to HOST_HOME_PERMISSION: open /home/vsts/.minikube/profiles/minikube/config.json: permission denied
* Suggestion: Your user lacks permissions to the minikube profile directory. Run: 'sudo chown -R $USER $HOME/.minikube; chmod -R u+wrx $HOME/.minikube' to fix
* Related issue: https://github.com/kubernetes/minikube/issues/9165
[...]

Issue Links

• Discovered while testing: FLINK-30893 Replace not specifying a driver in Minikube by something else
• is related to: FLINK-29671 Kubernetes e2e test fails during test initialization
• links to: GitHub Pull Request #21834
• links to: GitHub Pull Request #21836
• links to: GitHub Pull Request #21837