[SPARK-34738] Upgrade Minikube and kubernetes and move to docker 'virtualization' layer

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Component/s: jenkins, Kubernetes
• Assignee: Shane Knapp
• Reporter: Attila Zsolt Piros
• Created: 14/Mar/21 08:17
• Updated: 18/May/21 03:56
• Resolved: 17/May/21 23:12

Description

shaneknapp as we discussed on the mailing list Minikube can be upgraded to the latest (v1.18.1) and kubernetes version should be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).

Here is my PR which uses a new method to configure the kubernetes client. Thanks in advance to use it for testing on the Jenkins after the Minikube version is updated.

 

Added by Shane:

we also need to move from the kvm2 virtualization layer to docker.  docker is a recommended driver w/the latest versions of minikube, and this will allow devs to more easily recreate the minikube/k8s env on their local workstations and run the integration tests in an identical environment as jenkins.

the TL;DR is that upgrading to docker works, except that the PV integration tests are failing due to a couple of possible reasons:

1) the 'spark-kubernetes-driver' isn't properly being loaded (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)

2) during the PV test run, the error message 'Given path (/opt/spark/pv-tests/tmp4595937990978494271.txt) does not exist' shows up in the logs.  however, the mk cluster does mount successfully to the local bare-metal filesystem and if i 'minikube ssh' in to it, i can see the mount and read/write successfully to it (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312548&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312548)

i could really use some help, and if it's useful, i can create some local accounts manually and allow ssh access for a couple of people to assist me.