[BEAM-3821] Support a pluggable key management system (KMS)

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: sdk-ideas
• Assignee: Unassigned
• Reporter: Luke Cwik
• Created: 09/Mar/18 17:48
• Updated: 25/Jun/20 17:24

Description

Different runners want to have access to various cloud providers such as AWS, Azure, GCP yet still provide users the ability to have control over access to those resources.

Adding support for a pluggable key management solution would allow users to have control over which resources are accessed and when.

Some concrete scenarios would be:

• Access S3 and GCS from a Yarn cluster running on AWS.
• Control S3 resource access for pipelines executing on Google Cloud Dataflow

Issue Links

• Dependent: BEAM-5959 Add Cloud KMS support to GCS copies