[SPARK-25742] Is there a way to pass the Azure blob storage credentials to the spark for k8s init-container?

• Type: Question
• Status: Resolved
• Priority: Minor
• Resolution: Invalid
• Affects Version/s: 2.3.2
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Oscar Bonilla
• Created: 16/Oct/18 11:03
• Updated: 12/Dec/22 18:10
• Resolved: 17/Oct/18 08:52

Description

I'm trying to run spark on a kubernetes cluster in Azure. The idea is to store the Spark application jars and dependencies in a container in Azure Blob Storage.

I've tried to do this with a public container and this works OK, but when having a private Blob Storage container, the spark-init init container doesn't download the jars.

The equivalent in AWS S3 is as simple as adding the key_id and secret as environment variables, but I don't see how to do this for Azure Blob Storage.