[SPARK-30626] Add SPARK_APPLICATION_ID into driver pod env

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core
• Assignee: Jiaxin Shan
• Reporter: Jiaxin Shan
• Created: 23/Jan/20 19:32
• Updated: 17/May/20 18:23
• Resolved: 24/Jan/20 20:01

Description

This should be a minor improvement.

The use case is we want to look up environment variables and create application folder and redirect driver logs to application folder.  Executors has it and we want to make a change to driver as well. 

 

Limits:
 cpu: 1024m
 memory: 896Mi
 Requests:
 cpu: 1
 memory: 896Mi
Environment:
 SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP)
 SPARK_LOCAL_DIRS: /var/data/spark-9c315655-aba4-47fb-821c-30268d02af7e
 SPARK_CONF_DIR: /opt/spark/conf

 

https://github.com/apache/spark/blob/afe70b3b5321439318a456c7d19b7074171a286a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala#L73-L79

We need SPARK_APPLICATION_ID inside the pod to organize logs 

Issue Links

• links to: GitHub Pull Request #27347