[SPARK-49731] Support K8s volume `mount.subPathExpr` and `hostPath` volume `type`

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: k8s, Kubernetes
• Labels: pull-request-available
• Assignee: Enrico Minack Assign to me
• Reporter: Enrico Minack
• Created: 20/Sep/24 08:15
• Updated: 25/Sep/24 14:54
• Resolved: 25/Sep/24 14:53

Description

Spark on K8S supports mounting volumes to driver and executor pods. It supports the subPath but not the subPathExpr K8S option, which allows using environment variables for the subPath. Further, hostPath volumes are always mounted as default type "". Providing a different hostPath volume type allows for more flexibility.

Add the following configuration options:

• spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.subPathExpr
• spark.kubernetes.executor.volumes.hostPath.[VolumeName].options.type

Issue Links

• links to: GitHub Pull Request #48181