In Kubernetes, dynamic pod metadata can be set as environment variables and then used as substitution in other environment variables.
Currently in Airflow, the pod request creation set those variables after the ones that can use substitutions so them doen't do it.
Example usage is:
- define a variable SPARK_LOCAL_IP using the pod metadata status.podIP
- using this variable in Spark options as -Dspark.driver.host=$(SPARK_LOCAL_IP)
I will do a PR to fix this behavior.