[YARN-8071] Add ability to specify nodemanager environment variables individually

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.2.0
• Component/s: yarn
• Hadoop Flags: Reviewed
• Assignee: Jim Brennan
• Reporter: Jim Brennan
• Created: 23/Mar/18 20:59
• Updated: 16/Oct/19 22:00
• Resolved: 16/May/18 21:24

Description

YARN-6830 describes a problem where environment variables that contain commas cannot be specified via -Dmapreduce.map.env.

For example:

-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"

will set MOUNTS to /tmp/foo

In that Jira, aw suggested that we change the API to provide a way to specify environment variables individually, the same way that Spark does.

Rather than fight with a regex why not redefine the API instead?

 

-Dmapreduce.map.env.MODE=bar
-Dmapreduce.map.env.IMAGE_NAME=foo
-Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar

...

e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar

This greatly simplifies the input validation needed and makes it clear what is actually being defined.

The mapreduce properties were dealt with in MAPREDUCE-7069. This Jira will address the YARN properties.

Issue Links

• is related to: YARN-9906 When setting multi volumes throurh the "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS" setting is not valid
• relates to: YARN-8029 YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS should not use commas as separators
• relates to: YARN-6830 Support quoted strings for environment variables
• relates to: MAPREDUCE-7069 Add ability to specify user environment variables individually