[SPARK-33175] Detect duplicated mountPath and fail at Spark side

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.7, 3.0.2, 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Kubernetes
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 18/Oct/20 05:13
• Updated: 18/Oct/20 17:01
• Resolved: 18/Oct/20 17:01

Description

If there is a mountPath conflict, the pod is created and repeats the following error messages and keep running. This should not keep running and we had better fail at Spark side.

$ k get pod -l 'spark-role in (driver,executor)'
NAME    READY   STATUS    RESTARTS   AGE
tpcds   1/1     Running   0          14m

20/10/18 05:09:26 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber.
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: ...
Message: Pod "tpcds-exec-1" is invalid: spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/data1": must be unique.

Issue Links

• links to: [Github] Pull Request #30084 (dongjoon-hyun)