[SPARK-24028] [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 2.3.0
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Matt Cheah
• Created: 19/Apr/18 21:38
• Updated: 17/May/20 18:26
• Resolved: 20/Apr/18 00:26

Description

Currently we create the Kubernetes resources the driver depends on - such as the properties config map and secrets to mount into the pod - only after we create the driver pod. This is because we want these extra objects to immediately have an owner reference to be tied to the driver pod.

On our Kubernetes 1.9.4. cluster, we're seeing that sometimes this works fine, but other times the driver ends up being started with empty volumes instead of volumes with the contents of the secrets we expect. The result is that sometimes the driver will start without these files mounted, which leads to various failures if the driver requires these files to be present early on in their code. Missing the properties file config map, for example, would mean spark-submit doesn't have a properties file to read at all. See the warning on https://kubernetes.io/docs/concepts/storage/volumes/#secret.

Unfortunately we cannot link owner references to non-existent objects, so we have to do this instead:

1. Create the auxiliary resources without any owner references.
2. Create the driver pod mounting these resources into volumes, as before.
3. If #2 fails, clean up the resources created in #1.
4. Edit the auxiliary resources to have an owner reference for the driver pod.

The multi-step approach leaves a small chance for us to leak resources - for example, if we fail to make the resource edits in #4 for some reason. This also changes the permissioning mode required for spark-submit - credentials provided to spark-submit need to be able to edit resources in addition to creating them.