[SPARK-26239] Add configurable auth secret source in k8s backend

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core
• Assignee: Matt Cheah Assign to me
• Reporter: Marcelo Masiero Vanzin
• Created: 30/Nov/18 19:45
• Updated: 17/May/20 18:25
• Resolved: 11/Dec/18 21:51

Description

This is a follow up to SPARK-26194, which aims to add auto-generated secrets similar to the YARN backend.

There's a desire to support different ways to generate and propagate these auth secrets (e.g. using things like Vault). Need to investigate:

• exposing configuration to support that
• changing SecurityManager so that it can delegate some of the secret-handling logic to custom implementations
• figuring out whether this can also be used in client-mode, where the driver is not created by the k8s backend in Spark.

Issue Links

• relates to: SPARK-26194 Support automatic spark.authenticate secret in Kubernetes backend
• links to: [Github] Pull Request #23252 (mccheah)
• links to: GitHub Pull Request #23252