[SPARK-26301] Consider switching from putting secret in environment variable directly to using secret reference

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Matt Cheah
• Created: 07/Dec/18 06:02
• Updated: 17/May/20 18:26

Description

In SPARK-26194 we proposed using an environment variable that is loaded in the executor pod spec to share the generated SASL secret key between the driver and the executors. However in practice this is very difficult to secure. Most traditional Kubernetes deployments will handle permissions by allowing wide access to viewing pod specs but restricting access to view Kubernetes secrets. Now however any user that can view the pod spec can also view the contents of the SASL secrets.

An example use case where this quickly breaks down is in the case where a systems administrator is allowed to look at pods that run user code in order to debug failing infrastructure, but the cluster administrator should not be able to view contents of secrets or other sensitive data from Spark applications run by their users.

We propose modifying the existing solution to instead automatically create a Kubernetes Secret object containing the SASL encryption key, then using the secret reference feature in Kubernetes to store the data in the environment variable without putting the secret data in the pod spec itself.

Issue Links

• relates to: SPARK-26194 Support automatic spark.authenticate secret in Kubernetes backend