When a cluster has PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables set (needed for using non-system Python e.g. /usr/bin/anaconda/bin/python), then you are unable to override this per submission in YARN cluster mode.
When using spark-submit (in this case via LIVY) to submit with an override:
spark-submit --master yarn --deploy-mode cluster --conf 'spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=python3' --conf' 'spark.yarn.appMasterEnv.PYSPARK_PYTHON=python3' probe.py
the environment variable values will override the conf settings. A workaround for some can be to unset the env vars but that is not always possible (e.g. submitting batch via LIVY where you can only pass through the parameters to spark-submit).
Expectation is that the conf values above override the environment variables.
Fix is to change the order of application of conf and env vars in the yarn client.
Backporting this to 1.6 would be great and unblocking for me.