Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8071

Add ability to specify nodemanager environment variables individually

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.2.0
    • Component/s: yarn
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      YARN-6830 describes a problem where environment variables that contain commas cannot be specified via -Dmapreduce.map.env.

      For example:

      -Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"

      will set MOUNTS to /tmp/foo

      In that Jira, Allen Wittenauer suggested that we change the API to provide a way to specify environment variables individually, the same way that Spark does.

      Rather than fight with a regex why not redefine the API instead?

       

      -Dmapreduce.map.env.MODE=bar
      -Dmapreduce.map.env.IMAGE_NAME=foo
      -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar

      ...

      e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar

      This greatly simplifies the input validation needed and makes it clear what is actually being defined.

      The mapreduce properties were dealt with in MAPREDUCE-7069. This Jira will address the YARN properties.

        Attachments

        1. YARN-8071.001.patch
          9 kB
          Jim Brennan
        2. YARN-8071.002.patch
          9 kB
          Jim Brennan
        3. YARN-8071.003.patch
          12 kB
          Jim Brennan

          Issue Links

            Activity

              People

              • Assignee:
                Jim_Brennan Jim Brennan
                Reporter:
                Jim_Brennan Jim Brennan
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: