Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.4.1, 2.5.2, 2.7.2, 2.6.4, 2.8.1
-
None
Description
Given the default config of "yarn.nodemanager.env-whitelist", application should be able to set the env variable $HADOOP_CONF_DIR to value other than the one in the NodeManager system environment. However, I believe due to a bug in the org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch class, it is not possible so.
From the sanitizeEnv() method in the ContainerLaunch class (https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L977)
putEnvIfNotNull(environment,
Environment.HADOOP_CONF_DIR.name(),
System.getenv(Environment.HADOOP_CONF_DIR.name())
);
if (!Shell.WINDOWS) {
environment.put("JVM_PID", "$$");
}
String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST, YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
for(String whitelistEnvVariable : whitelist) {
putEnvIfAbsent(environment, whitelistEnvVariable.trim());
}
...
private static void putEnvIfAbsent(
Map<String, String> environment, String variable) {
if (environment.get(variable) == null) {
putEnvIfNotNull(environment, variable, System.getenv(variable));
}
}
So there two issues here.
1. the environment is already set with the system environment of the NM in the putEnvIfNotNull call, hence the putEnvIfAbsent call will never set it to some new value
2. Inside the putEnvIfAbsent call, it uses the system environment of the NM, which it should be using the one from the launchContext instead.
Attachments
Attachments
Issue Links
- is required by
-
-
- Patch Available
-