Given the default config of "yarn.nodemanager.env-whitelist", application should be able to set the env variable $HADOOP_CONF_DIR to value other than the one in the NodeManager system environment. However, I believe due to a bug in the org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch class, it is not possible so.
From the sanitizeEnv() method in the ContainerLaunch class (https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L977)
So there two issues here.
1. the environment is already set with the system environment of the NM in the putEnvIfNotNull call, hence the putEnvIfAbsent call will never set it to some new value
2. Inside the putEnvIfAbsent call, it uses the system environment of the NM, which it should be using the one from the launchContext instead.