Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5877

Allow all env's from yarn.nodemanager.env-whitelist to get overridden during launch

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As per the yarn.nodemanager.env-whitelist for the configured values should containers may override rather than use NodeManager's default.

        <property>
          <description>Environment variables that containers may override rather than use NodeManager's default.</description>
          <name>yarn.nodemanager.env-whitelist</name>
          <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME</value>
        </property>
      
      

      But only the following containers can override

          whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
          whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
          whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
          whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
          whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
      
      1. 5877 nm logs of test.log
        886 kB
        Bibin A Chundatt
      2. bootstrap.sh
        0.7 kB
        Bibin A Chundatt
      3. Dockerfile
        3 kB
        Bibin A Chundatt
      4. YARN-5877.0001.patch
        2 kB
        Bibin A Chundatt
      5. YARN-5877.0002.patch
        5 kB
        Bibin A Chundatt
      6. YARN-5877.0003.patch
        6 kB
        Bibin A Chundatt
      7. YARN-5877.0004.patch
        5 kB
        Bibin A Chundatt
      8. YARN-5877.0005.patch
        8 kB
        Bibin A Chundatt
      9. YARN-5877.0006.patch
        8 kB
        Bibin A Chundatt
      10. yarn-site.xml
        6 kB
        Bibin A Chundatt

        Issue Links

          Activity

          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          As per my understanding we should change as following

          -    whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
          -    whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
          +    String[] nmWhiteList = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
          +        YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
          +    for (String param : nmWhiteList) {
          +      whitelist.add(param);
          +    }
          
          

          Thoughts??

          Show
          bibinchundatt Bibin A Chundatt added a comment - As per my understanding we should change as following - whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name()); - whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name()); - whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name()); - whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name()); - whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name()); + String [] nmWhiteList = conf.get(YarnConfiguration.NM_ENV_WHITELIST, + YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split( "," ); + for ( String param : nmWhiteList) { + whitelist.add(param); + } Thoughts??
          Hide
          varun_saxena Varun Saxena added a comment -

          Bibin A Chundatt, we do take NM whitelist from configuration in ContainerLaunch#sanitizeEnv. So IIUC, changes mentioned in the preceding comment are not required.

              // allow containers to override these variables
              String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST, YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
              
              for(String whitelistEnvVariable : whitelist) {
                putEnvIfAbsent(environment, whitelistEnvVariable.trim());
              }
          

          Are you saying why are we adding these specific environment variables by default even though the configuration says otherwise ?

          Show
          varun_saxena Varun Saxena added a comment - Bibin A Chundatt , we do take NM whitelist from configuration in ContainerLaunch#sanitizeEnv. So IIUC, changes mentioned in the preceding comment are not required. // allow containers to override these variables String [] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST, YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split( "," ); for ( String whitelistEnvVariable : whitelist) { putEnvIfAbsent(environment, whitelistEnvVariable.trim()); } Are you saying why are we adding these specific environment variables by default even though the configuration says otherwise ?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Varun Saxena
          What i am saying is the whitelist doen't even apply based on yarn.nodemanager.env-whitelist

          ContainerExecutor#writeLaunchEnv

            @VisibleForTesting
            public void writeLaunchEnv(OutputStream out, Map<String, String> environment,
                Map<Path, List<String>> resources, List<String> command, Path logDir,
                String user, String outFilename) throws IOException {
          ....
              Set<String> whitelist = new HashSet<>();
          .....
              if (environment != null) {
                for (Map.Entry<String, String> env : environment.entrySet()) {
                  if (!whitelist.contains(env.getKey())) {
                    sb.env(env.getKey(), env.getValue());
                  } else {
                    sb.whitelistedEnv(env.getKey(), env.getValue());
                  }
                }
              }
          

          Here we are only replacing the list which is created new .

          Show
          bibinchundatt Bibin A Chundatt added a comment - Varun Saxena What i am saying is the whitelist doen't even apply based on yarn.nodemanager.env-whitelist ContainerExecutor#writeLaunchEnv @VisibleForTesting public void writeLaunchEnv(OutputStream out, Map< String , String > environment, Map<Path, List< String >> resources, List< String > command, Path logDir, String user, String outFilename) throws IOException { .... Set< String > whitelist = new HashSet<>(); ..... if (environment != null ) { for (Map.Entry< String , String > env : environment.entrySet()) { if (!whitelist.contains(env.getKey())) { sb.env(env.getKey(), env.getValue()); } else { sb.whitelistedEnv(env.getKey(), env.getValue()); } } } Here we are only replacing the list which is created new .
          Hide
          varun_saxena Varun Saxena added a comment -

          Hmmm...You are correct.
          The behavior seems to have been changed in YARN-3853. Let me check why.

          Show
          varun_saxena Varun Saxena added a comment - Hmmm...You are correct. The behavior seems to have been changed in YARN-3853 . Let me check why.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -
          Show
          bibinchundatt Bibin A Chundatt added a comment - cc / Varun Vasudev
          Hide
          vvasudev Varun Vasudev added a comment -

          The variables should be overriden - ContainerLaunch#sanitizeEnv should take care of it no?

              // allow containers to override these variables
              String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST, YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
              
              for(String whitelistEnvVariable : whitelist) {
                putEnvIfAbsent(environment, whitelistEnvVariable.trim());
              }
          
          Show
          vvasudev Varun Vasudev added a comment - The variables should be overriden - ContainerLaunch#sanitizeEnv should take care of it no? // allow containers to override these variables String [] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST, YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split( "," ); for ( String whitelistEnvVariable : whitelist) { putEnvIfAbsent(environment, whitelistEnvVariable.trim()); }
          Hide
          varun_saxena Varun Saxena added a comment -

          Looked at the code. yarn.nodemanager.env-whitelist is originally meant to allow certain environment variables to be taken from container launch context instead of NM system environment (if available) and this, from code, seems to work fine even with the changes made in YARN-3853. We will put value from NM evnironment only if its not available in container launch context which the code pointed out above does.

          The code added in YARN-3853 is meant to override the environment variables pointed out by you, further. Probably for docker. The variable name is whitelist but that should not be confused with NM whitelisting. Should we allow similar behavior for other environment variables too i.e. make it a configurable list similar to NM whitelist or use NM whitelist ? Well possibly it can be, but I am not really sure of the implications.

          Show
          varun_saxena Varun Saxena added a comment - Looked at the code. yarn.nodemanager.env-whitelist is originally meant to allow certain environment variables to be taken from container launch context instead of NM system environment (if available) and this, from code, seems to work fine even with the changes made in YARN-3853 . We will put value from NM evnironment only if its not available in container launch context which the code pointed out above does. The code added in YARN-3853 is meant to override the environment variables pointed out by you, further. Probably for docker. The variable name is whitelist but that should not be confused with NM whitelisting. Should we allow similar behavior for other environment variables too i.e. make it a configurable list similar to NM whitelist or use NM whitelist ? Well possibly it can be, but I am not really sure of the implications.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thank you Varun Saxena and Varun Vasudev

          During discussion in MAPREDUCE-6704 we had encountered a scenario were white listing nm-env during container launch could help.

          Make it a configurable list similar to NM whitelist or use NM whitelist

          IIUC using NM whitelist shouldn't cause any issue.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Thank you Varun Saxena and Varun Vasudev During discussion in MAPREDUCE-6704 we had encountered a scenario were white listing nm-env during container launch could help. Make it a configurable list similar to NM whitelist or use NM whitelist IIUC using NM whitelist shouldn't cause any issue.
          Hide
          sunilg Sunil G added a comment -

          Currently whitelistedEnv is used to emit special ENV variables which are to be with container context. Post YARN-3853, its limited to a set of ENV variables such as below.

              whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
              whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
              whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
              whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
              whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
          

          So if a user adds HADOOP_MAPRED_HOME to whitelist via yarn.nodemanager.env-whitelist then it will NOT be emitted with whitelistedEnv. It will be just set as a normal env variable.
          Here if user wants all such variables from yarn.nodemanager.env-whitelist to emit with help of whitelistedEnv api to launch_container.sh, then proposed solution makes sense to me. But I am not very sure whether all such variables configured under yarn.nodemanager.env-whitelist is to be emitted with help of whitelistedEnv api. For eg CLASSPATH_PREPEND_DISTCACHE.

          Varun Vasudev , thoughts?

          Show
          sunilg Sunil G added a comment - Currently whitelistedEnv is used to emit special ENV variables which are to be with container context. Post YARN-3853 , its limited to a set of ENV variables such as below. whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name()); whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name()); whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name()); whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name()); whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name()); So if a user adds HADOOP_MAPRED_HOME to whitelist via yarn.nodemanager.env-whitelist then it will NOT be emitted with whitelistedEnv . It will be just set as a normal env variable. Here if user wants all such variables from yarn.nodemanager.env-whitelist to emit with help of whitelistedEnv api to launch_container.sh , then proposed solution makes sense to me. But I am not very sure whether all such variables configured under yarn.nodemanager.env-whitelist is to be emitted with help of whitelistedEnv api. For eg CLASSPATH_PREPEND_DISTCACHE . Varun Vasudev , thoughts?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thank you Sunil G for looking into issue.

          But I am not very sure whether all such variables configured under yarn.nodemanager.env-whitelist is to be emitted with help of whitelistedEnv api

          IMHO doesn't impact current flow. If CLASSPATH_PREPEND_DISTCACHE is available in system env or launchContext will export same value in launch_container as default i.e.

          export CLASSPATH_PREPEND_DISTCACHE=${CLASSPATH_PREPEND_DISTCACHE:-" value from env or launchContext  "}
          
          Show
          bibinchundatt Bibin A Chundatt added a comment - Thank you Sunil G for looking into issue. But I am not very sure whether all such variables configured under yarn.nodemanager.env-whitelist is to be emitted with help of whitelistedEnv api IMHO doesn't impact current flow. If CLASSPATH_PREPEND_DISTCACHE is available in system env or launchContext will export same value in launch_container as default i.e. export CLASSPATH_PREPEND_DISTCACHE=${CLASSPATH_PREPEND_DISTCACHE:- " value from env or launchContext " }
          Hide
          sunilg Sunil G added a comment -

          Thanks Bibin A Chundatt.
          I think this makes sense to me. I am still thinking about cases like rolling upgrade. if there are some custom variables exported by nm_whilelist earlier, from now onwards these variables will try to get it from system env. Ideally a mismatch may occur only from docker point of view where env and launchcontext may differ. So i think there need to be some sanity tests to be done in Docker too for MR and another simple app. Could you pls help to verify that.

          Show
          sunilg Sunil G added a comment - Thanks Bibin A Chundatt . I think this makes sense to me. I am still thinking about cases like rolling upgrade. if there are some custom variables exported by nm_whilelist earlier, from now onwards these variables will try to get it from system env. Ideally a mismatch may occur only from docker point of view where env and launchcontext may differ. So i think there need to be some sanity tests to be done in Docker too for MR and another simple app. Could you pls help to verify that.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G

          Ideally a mismatch may occur only from docker point of view where env and launchcontext may differ.

          Have a query regarding the same. In case there is mismatch its supposed to using env only rt? How about adding new config such as container-launch-whitelist instead of using the same NM whitelist to provide flexibility.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G Ideally a mismatch may occur only from docker point of view where env and launchcontext may differ. Have a query regarding the same. In case there is mismatch its supposed to using env only rt? How about adding new config such as container-launch-whitelist instead of using the same NM whitelist to provide flexibility.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching dockefile and bootstrap file which i tried out to create the docker image in ubuntu. Using local tar file and jdk based on https://github.com/sequenceiq/hadoop-docker

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching dockefile and bootstrap file which i tried out to create the docker image in ubuntu. Using local tar file and jdk based on https://github.com/sequenceiq/hadoop-docker
          Hide
          varun_saxena Varun Saxena added a comment -

          Bibin A Chundatt,
          I was leaning towards the same. Better to have a new configuration. Because intention of NM whitelist configuration is different.

          Show
          varun_saxena Varun Saxena added a comment - Bibin A Chundatt , I was leaning towards the same. Better to have a new configuration. Because intention of NM whitelist configuration is different.
          Hide
          sunilg Sunil G added a comment -

          Yes. This is an option, however it will be difficult for an admin to know which configuration to use at different contexts. Yes, we can make it more easier by providing a better config name / documentation. I am not so sure how effective it may be and whether users may tend to use one in all cases.

          I think current implementation can go wrong only if there is mismatch with System.env and NM context.. Usually for any normal launchers, both will be same. For docker, System env looks correctly sanitized as of now. If some env's are going wrong post this ticket, it must be because of the mismatch b/w system.env and NM context.

          I am fine with both approaches here, but would like to here more from other folks.

          Show
          sunilg Sunil G added a comment - Yes. This is an option, however it will be difficult for an admin to know which configuration to use at different contexts. Yes, we can make it more easier by providing a better config name / documentation. I am not so sure how effective it may be and whether users may tend to use one in all cases. I think current implementation can go wrong only if there is mismatch with System.env and NM context.. Usually for any normal launchers, both will be same. For docker, System env looks correctly sanitized as of now. If some env's are going wrong post this ticket, it must be because of the mismatch b/w system.env and NM context. I am fine with both approaches here, but would like to here more from other folks.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch for the same.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch for the same.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 44s trunk passed
          +1 compile 0m 36s trunk passed
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 37s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 0m 52s trunk passed
          +1 javadoc 0m 22s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 31s the patch passed
          +1 javac 0m 31s the patch passed
          -0 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 mvnsite 0m 33s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 1s the patch passed
          +1 javadoc 0m 18s the patch passed
          -1 unit 14m 15s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          31m 39s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5877
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839375/YARN-5877.0001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6bb71e5fa2be 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b2d4b7b
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13955/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13955/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13955/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13955/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 44s trunk passed +1 compile 0m 36s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 0m 52s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed -0 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 14m 15s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 31m 39s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839375/YARN-5877.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6bb71e5fa2be 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b2d4b7b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13955/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13955/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13955/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13955/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch after updating test cases.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch after updating test cases.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 58s trunk passed
          +1 compile 0m 27s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 26s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 40s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 24s the patch passed
          +1 javac 0m 24s the patch passed
          +1 checkstyle 0m 14s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 45s the patch passed
          +1 javadoc 0m 14s the patch passed
          +1 unit 13m 20s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          27m 6s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5877
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839678/YARN-5877.0002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c0ef740a5969 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7584fbf
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13978/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13978/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 58s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 40s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 45s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 13m 20s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 27m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839678/YARN-5877.0002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c0ef740a5969 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7584fbf Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13978/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13978/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment - - edited

          Is this the right thing to do ?

          NM whitelist configuration is used to allow containers to override NM's default environment.
          Let us assume there is an environment variable DUMMY_ENV. Assume in some deployment, for some jobs, its set in the ContainerLaunchContext and for all other jobs we want a default which can be set in NM environment.
          Before the change in the patch, we could have set DUMMY_ENV in NM whitelist and NM would have picked up value for this environment variable from container launch context (if available) otherwise taken it from NM environment.
          After the change in this patch, NM would keep the value coming from container launch context as default and use that only if the environment variable on NM is not set.

          This is a departure from current behavior. Probably you can argue none of us are using it this way but its still a change in behavior.
          We have done the same for HADOOP_YARN_HOME, HADOOP_COMMON_HOME, etc. in YARN-3853 but these are well known environment variables so probably nobody uses them the way I mentioned above. Atleast we do not. So it may be safe to use them this way.

          If we were to have a different configuration, we can probably get rid of these concerns.

          Thoughts ?

          Show
          varun_saxena Varun Saxena added a comment - - edited Is this the right thing to do ? NM whitelist configuration is used to allow containers to override NM's default environment. Let us assume there is an environment variable DUMMY_ENV. Assume in some deployment, for some jobs, its set in the ContainerLaunchContext and for all other jobs we want a default which can be set in NM environment. Before the change in the patch, we could have set DUMMY_ENV in NM whitelist and NM would have picked up value for this environment variable from container launch context (if available) otherwise taken it from NM environment. After the change in this patch, NM would keep the value coming from container launch context as default and use that only if the environment variable on NM is not set. This is a departure from current behavior. Probably you can argue none of us are using it this way but its still a change in behavior. We have done the same for HADOOP_YARN_HOME, HADOOP_COMMON_HOME, etc. in YARN-3853 but these are well known environment variables so probably nobody uses them the way I mentioned above. Atleast we do not. So it may be safe to use them this way. If we were to have a different configuration, we can probably get rid of these concerns. Thoughts ?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Before the change in the patch, we could have set DUMMY_ENV in NM whitelist and NM would have picked up value for this environment variable from container launch context (if available) otherwise taken it from NM environment.
          After the change in this patch, NM would keep the value coming from container launch context as default and use that only if the environment variable on NM is not set.

          Could please recheck this as per current Shell launch the system env will not be inherited from parent to child process. So will thr be a case other than docker this could happen??

          Show
          bibinchundatt Bibin A Chundatt added a comment - Before the change in the patch, we could have set DUMMY_ENV in NM whitelist and NM would have picked up value for this environment variable from container launch context (if available) otherwise taken it from NM environment. After the change in this patch, NM would keep the value coming from container launch context as default and use that only if the environment variable on NM is not set. Could please recheck this as per current Shell launch the system env will not be inherited from parent to child process. So will thr be a case other than docker this could happen??
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          I am open to adding new configuration as mentioned earlier.

          Show
          bibinchundatt Bibin A Chundatt added a comment - I am open to adding new configuration as mentioned earlier.
          Hide
          varun_saxena Varun Saxena added a comment -

          Sorry for the confusion. Correct. We do not inherit parent environment while launching the container. We explicitly clear the environment in ProcessBuilder and use the environment in container launch context instead.

          But can a similar thing happen in docker scenario ? The main thing which I wanted to highlight was that we should probably give preference to environment set in container launch context if its also configured in the whitelist no matter what the environment is set to.

          Show
          varun_saxena Varun Saxena added a comment - Sorry for the confusion. Correct. We do not inherit parent environment while launching the container. We explicitly clear the environment in ProcessBuilder and use the environment in container launch context instead. But can a similar thing happen in docker scenario ? The main thing which I wanted to highlight was that we should probably give preference to environment set in container launch context if its also configured in the whitelist no matter what the environment is set to.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          For HADOOP_MAPRED_HOME this behaviour would be same since earlier also the value was fetched from ENV. So to clear the confusions it would be better to solve with new properly yarn.nodemanager.container-launch-env-whitelist which would provide more flexibility .

          Show
          bibinchundatt Bibin A Chundatt added a comment - For HADOOP_MAPRED_HOME this behaviour would be same since earlier also the value was fetched from ENV. So to clear the confusions it would be better to solve with new properly yarn.nodemanager.container-launch-env-whitelist which would provide more flexibility .
          Hide
          sunilg Sunil G added a comment -

          Bibin A Chundatt and Varun Saxena

          Ideally there will be difference with NM context and system ENV only in Docker scenario.

          I am not against new configuration. But how can user know which config to use. In case of normal launch, there are no difference with these 2 configurations. Only in docker, there are difference. So i think are we going a case which is only for docker? if so, configuration also need to be docker specific

          Thoughts?

          Show
          sunilg Sunil G added a comment - Bibin A Chundatt and Varun Saxena Ideally there will be difference with NM context and system ENV only in Docker scenario. I am not against new configuration. But how can user know which config to use. In case of normal launch, there are no difference with these 2 configurations. Only in docker, there are difference. So i think are we going a case which is only for docker? if so, configuration also need to be docker specific Thoughts?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G. Attaching patch based on separate parameter. Only when container env is of type docker. Hope this patch is inline with your suggestion of docker specific too.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G . Attaching patch based on separate parameter. Only when container env is of type docker. Hope this patch is inline with your suggestion of docker specific too.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 7m 26s trunk passed
          +1 compile 5m 13s trunk passed
          +1 checkstyle 0m 48s trunk passed
          +1 mvnsite 1m 15s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 2m 14s trunk passed
          +1 javadoc 0m 59s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 0s the patch passed
          +1 compile 6m 2s the patch passed
          +1 javac 6m 2s the patch passed
          -0 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 208 unchanged - 1 fixed = 211 total (was 209)
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 45s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 34s the patch passed
          +1 javadoc 0m 55s the patch passed
          -1 unit 0m 38s hadoop-yarn-api in the patch failed.
          +1 unit 14m 48s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          57m 21s



          Reason Tests
          Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5877
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840288/YARN-5877.0003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 22dfd93974ee 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3541ed8
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14061/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14061/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14061/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14061/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 7m 26s trunk passed +1 compile 5m 13s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 1m 15s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 2m 14s trunk passed +1 javadoc 0m 59s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 6m 2s the patch passed +1 javac 6m 2s the patch passed -0 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 208 unchanged - 1 fixed = 211 total (was 209) +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 34s the patch passed +1 javadoc 0m 55s the patch passed -1 unit 0m 38s hadoop-yarn-api in the patch failed. +1 unit 14m 48s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 57m 21s Reason Tests Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840288/YARN-5877.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 22dfd93974ee 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3541ed8 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14061/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14061/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14061/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14061/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          varun_saxena Varun Saxena added a comment -

          Thanks Bibin A Chundatt for the patch. The approach as such seems fine and is consistent with previous implementation. Do we however need to provide a facility for container launch context env to take precedence for some containers and take default from docker env for some others ? Anyways, that would need more discussion regarding if there is any use case for this.

          Coming to the patch.

          1. We need to add this config in yarn-default.xml
          2. getWhitelistEnv can be protected
          3. Name getWhitelistEnv -> getEnvToBeWhitelisted ? Also javadoc should clearly indicate what this method is meant for and why derived classes should override it.
          4. There is an unused import in ContainerExecutor.
          5. In LinuxContainerExecutor#getWhitelistEnv, while iterating over the envs configured, we should trim them before adding them to the set.

          I would like to hear from guys who worked in the area of docker support in YARN as well.

          Show
          varun_saxena Varun Saxena added a comment - Thanks Bibin A Chundatt for the patch. The approach as such seems fine and is consistent with previous implementation. Do we however need to provide a facility for container launch context env to take precedence for some containers and take default from docker env for some others ? Anyways, that would need more discussion regarding if there is any use case for this. Coming to the patch. We need to add this config in yarn-default.xml getWhitelistEnv can be protected Name getWhitelistEnv -> getEnvToBeWhitelisted ? Also javadoc should clearly indicate what this method is meant for and why derived classes should override it. There is an unused import in ContainerExecutor. In LinuxContainerExecutor#getWhitelistEnv, while iterating over the envs configured, we should trim them before adding them to the set. I would like to hear from guys who worked in the area of docker support in YARN as well.
          Hide
          varun_saxena Varun Saxena added a comment -

          Moreover, maybe docker related logic can be moved to its relevant runtime class so that related code logic does not spread over multiple classes.

          Show
          varun_saxena Varun Saxena added a comment - Moreover, maybe docker related logic can be moved to its relevant runtime class so that related code logic does not spread over multiple classes.
          Hide
          varun_saxena Varun Saxena added a comment -

          Do we however need to provide a facility for container launch context env to take precedence for some containers and take default from docker env for some others ? Anyways, that would need more discussion

          As this is unreleased, so this should be fine. I think we can simply state what is supported and what is not.

          Show
          varun_saxena Varun Saxena added a comment - Do we however need to provide a facility for container launch context env to take precedence for some containers and take default from docker env for some others ? Anyways, that would need more discussion As this is unreleased, so this should be fine. I think we can simply state what is supported and what is not.
          Hide
          sunilg Sunil G added a comment -

          Varun Saxena and Bibin A Chundatt

          Do we however need to provide a facility for container launch context env to take precedence for some containers and take default from docker env for some others ? Anyways, that would need more discussion regarding if there is any use case for this.

          I think we are little early to reach to conclusion on this as we do not have any substantial use cases yet.

          docker-env-whitelist

          To add this new configuration, we have discussed the real causes in detail. And this patch is planning to use this as a general config and HADOOP_MAPRED_HOME is the only configuration which will be configured via this.
          I am still not very convinced with few possible mis uses.

          • How can we prevent user from configuring same config in both configs? If so which will get preference ?
          • I am still thinking of the explanation to be provided to users for the use of this new config? I still think it will be very tough to convince users with real usage.

          I am bouncing this question again since its the case of a new config to client side.

          Thoughts?

          Show
          sunilg Sunil G added a comment - Varun Saxena and Bibin A Chundatt Do we however need to provide a facility for container launch context env to take precedence for some containers and take default from docker env for some others ? Anyways, that would need more discussion regarding if there is any use case for this. I think we are little early to reach to conclusion on this as we do not have any substantial use cases yet. docker-env-whitelist To add this new configuration, we have discussed the real causes in detail. And this patch is planning to use this as a general config and HADOOP_MAPRED_HOME is the only configuration which will be configured via this. I am still not very convinced with few possible mis uses. How can we prevent user from configuring same config in both configs? If so which will get preference ? I am still thinking of the explanation to be provided to users for the use of this new config? I still think it will be very tough to convince users with real usage. I am bouncing this question again since its the case of a new config to client side. Thoughts?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          How can we prevent user from configuring same config in both configs? If so which will get preference ?

          So we really need to prevent? Since the property value we will never get from NM except for HADOOP_MAPRED_HOME??

          w.r.t to new property docker-env-whitelist it provides extra flexibility only for docker.

          Show
          bibinchundatt Bibin A Chundatt added a comment - How can we prevent user from configuring same config in both configs? If so which will get preference ? So we really need to prevent? Since the property value we will never get from NM except for HADOOP_MAPRED_HOME?? w.r.t to new property docker-env-whitelist it provides extra flexibility only for docker.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G
          Is the current approach fine??

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G Is the current approach fine??
          Hide
          varun_saxena Varun Saxena added a comment -

          If the scenario I pointed out above is something which we do not want to support, then a new config may not be necessary.
          What do you think Bibin A Chundatt ?

          Show
          varun_saxena Varun Saxena added a comment - If the scenario I pointed out above is something which we do not want to support, then a new config may not be necessary. What do you think Bibin A Chundatt ?
          Hide
          sunilg Sunil G added a comment -

          We have checked various cases for this issue in detail and had some offline discussions with Varun Vasudev and Bibin A Chundatt also.

          Now coming to the issue mentioned. MAPREDUCE-6704 has come in to a common consensus that HADOOP_MAPRED_HOME could be published with nm-white-list so that even if this variable is not present in env, we can take the configured value. This will solve the container launch problem mentioned there.

          As of today, if we publish HADOOP_MAPRED_HOME through nm-whitelist-env, it will look like below in launch_container.sh.
          export HADOOP_MAPRED_HOME="/home/hadoopbuild"

          As mentioned above, this will be fine in case of normal container launch. But in docker scenario, the path value given may not be valid. In such case, we will see the failure as mentioned in this jira. Ideally if this environment variable were published as whitelist env, then it could try to take from env itself first.

          export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/home/hadoopbuild"}
          

          Here container will try to get the value from HADOOP_MAPRED_HOME if its available in env first. If not, it will take value which is given as "/home/hadoopbuild". In case of docker, this value was set in its system env itself (i think Bibin used sequence iq docker image here which had this value).

          So in conclusion, all variables mentioned in yarn.nodemanager.env-whitelist could be whitelisted. This can solve the problem. If some docker images doesn't have this env variable set in its env, and if the default path to HADOOP_MAPRED_HOME is also invalid, then user has to set correct mapred home path in yarn.app.mapreduce.am.env.

          I think YARN-5877.0002.patch is good one to go. Bibin A Chundatt could you rebase the patch and give a latest version number to compile against trunk. If possible, could you please share test results with this latest patch too in docker.

          Show
          sunilg Sunil G added a comment - We have checked various cases for this issue in detail and had some offline discussions with Varun Vasudev and Bibin A Chundatt also. Now coming to the issue mentioned. MAPREDUCE-6704 has come in to a common consensus that HADOOP_MAPRED_HOME could be published with nm-white-list so that even if this variable is not present in env, we can take the configured value. This will solve the container launch problem mentioned there. As of today, if we publish HADOOP_MAPRED_HOME through nm-whitelist-env, it will look like below in launch_container.sh. export HADOOP_MAPRED_HOME="/home/hadoopbuild" As mentioned above, this will be fine in case of normal container launch. But in docker scenario, the path value given may not be valid. In such case, we will see the failure as mentioned in this jira. Ideally if this environment variable were published as whitelist env, then it could try to take from env itself first. export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:- "/home/hadoopbuild" } Here container will try to get the value from HADOOP_MAPRED_HOME if its available in env first. If not, it will take value which is given as "/home/hadoopbuild". In case of docker, this value was set in its system env itself (i think Bibin used sequence iq docker image here which had this value). So in conclusion, all variables mentioned in yarn.nodemanager.env-whitelist could be whitelisted. This can solve the problem. If some docker images doesn't have this env variable set in its env, and if the default path to HADOOP_MAPRED_HOME is also invalid, then user has to set correct mapred home path in yarn.app.mapreduce.am.env . I think YARN-5877 .0002.patch is good one to go. Bibin A Chundatt could you rebase the patch and give a latest version number to compile against trunk. If possible, could you please share test results with this latest patch too in docker.
          Hide
          templedf Daniel Templeton added a comment -

          Patch 2 looks fine to me.

          Show
          templedf Daniel Templeton added a comment - Patch 2 looks fine to me.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thank you Sunil G for summarizing discussion.
          Updating patch based on the offline discussion

          Show
          bibinchundatt Bibin A Chundatt added a comment - Thank you Sunil G for summarizing discussion. Updating patch based on the offline discussion
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 3s trunk passed
          +1 compile 0m 37s trunk passed
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 0m 52s trunk passed
          +1 javadoc 0m 23s trunk passed
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 32s the patch passed
          +1 javac 0m 32s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 33s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 58s the patch passed
          +1 javadoc 0m 19s the patch passed
          +1 unit 14m 24s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          32m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5877
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842266/YARN-5877.0004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 27223fe8168e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f54afdb
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14218/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14218/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 3s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 52s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 32s the patch passed +1 javac 0m 32s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 58s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 14m 24s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 32m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842266/YARN-5877.0004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 27223fe8168e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f54afdb Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14218/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14218/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Latest patch looks fine for me as well.
          Bibin A Chundatt, could you please help to share the test results with latest patch on docker env.

          Show
          sunilg Sunil G added a comment - Latest patch looks fine for me as well. Bibin A Chundatt , could you please help to share the test results with latest patch on docker env.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G

          Last time i tested it was working fine. Also Zhankun Tang has done as part of MAPREDUCE-6704 too.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G Last time i tested it was working fine. Also Zhankun Tang has done as part of MAPREDUCE-6704 too.
          Hide
          sunilg Sunil G added a comment -

          Thanks. I suppose that test is done with version 2 of patch, correct?

          Show
          sunilg Sunil G added a comment - Thanks. I suppose that test is done with version 2 of patch, correct?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G
          With same ContainerExecutor changes. Test was done with version1.
          Only difference between version 1 and version 2 is in test.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G With same ContainerExecutor changes. Test was done with version1. Only difference between version 1 and version 2 is in test.
          Hide
          sunilg Sunil G added a comment -

          Thanks Bibin A Chundatt for the confirmation.

          +1 from my end.
          I will commit this patch later today if there are no objections.

          Show
          sunilg Sunil G added a comment - Thanks Bibin A Chundatt for the confirmation. +1 from my end. I will commit this patch later today if there are no objections.
          Hide
          templedf Daniel Templeton added a comment -

          Latest patch is fine for me. Looking at it, though, it occurs to me that we should really be adding a test for the change in functionality. Any chance you can add a test for writeLaunchEnv() to TestContainerExecutor? We should also test the whole thing in MAPREDUCE-6704.

          Show
          templedf Daniel Templeton added a comment - Latest patch is fine for me. Looking at it, though, it occurs to me that we should really be adding a test for the change in functionality. Any chance you can add a test for writeLaunchEnv() to TestContainerExecutor ? We should also test the whole thing in MAPREDUCE-6704 .
          Hide
          sunilg Sunil G added a comment -

          In MAPREDUCE-6704, temp.patch was attached which is same as the latest patch attached in this jira. Bibin A Chundatt, please confirm that point.
          Also with that patch, Zhankun Tang has tested in a docker env and it was working fine as per last statement there.
          It seems Bibin A Chundatt also has tested patch 2 (same as patch 4) in a docker environment (with sequenceiq docker image).

          I agree with Daniel Templeton here, it will be a good chance to write this test case which was not there in first place. Bibin A Chundatt, as part of this test, i think we just need to ensure that the env variable is written as whitelisted env when supplied from yarn.nodemanager.env-whitelist

          Show
          sunilg Sunil G added a comment - In MAPREDUCE-6704 , temp.patch was attached which is same as the latest patch attached in this jira. Bibin A Chundatt , please confirm that point. Also with that patch, Zhankun Tang has tested in a docker env and it was working fine as per last statement there. It seems Bibin A Chundatt also has tested patch 2 (same as patch 4) in a docker environment (with sequenceiq docker image). I agree with Daniel Templeton here, it will be a good chance to write this test case which was not there in first place. Bibin A Chundatt , as part of this test, i think we just need to ensure that the env variable is written as whitelisted env when supplied from yarn.nodemanager.env-whitelist
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G

          Change in temp and 0001 are both same

          -    whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
          -    whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
          -    whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
          +    String[] nmWhiteList = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
          +        YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
          +    for (String param : nmWhiteList) {
          +      whitelist.add(param);
          +   
          

          Attaching patch with testcase

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G Change in temp and 0001 are both same - whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name()); - whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name()); - whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name()); - whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name()); - whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name()); + String [] nmWhiteList = conf.get(YarnConfiguration.NM_ENV_WHITELIST, + YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split( "," ); + for ( String param : nmWhiteList) { + whitelist.add(param); + Attaching patch with testcase
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 44s trunk passed
          +1 compile 0m 26s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 39s trunk passed
          +1 javadoc 0m 16s trunk passed
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 24s the patch passed
          +1 javac 0m 24s the patch passed
          -0 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 82 unchanged - 0 fixed = 83 total (was 82)
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 46s the patch passed
          +1 javadoc 0m 14s the patch passed
          +1 unit 12m 46s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          26m 11s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5877
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842946/YARN-5877.0005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 75b5cba7a8d2 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b0b033e
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14270/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14270/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14270/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 39s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -0 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 82 unchanged - 0 fixed = 83 total (was 82) +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 12m 46s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 26m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842946/YARN-5877.0005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 75b5cba7a8d2 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b0b033e Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14270/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14270/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14270/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Thanks Bibin A Chundatt for the updated patch.

          If testWriteEnvExport could also add one more variable to env, and that one need not have to add to nm-white-list . So this will be like a normal env variable. I think we do not have other test cases to cover these area. So it will be better to have this -ve cases also added to testWriteEnvExport itself.

          Show
          sunilg Sunil G added a comment - Thanks Bibin A Chundatt for the updated patch. If testWriteEnvExport could also add one more variable to env, and that one need not have to add to nm-white-list . So this will be like a normal env variable. I think we do not have other test cases to cover these area. So it will be better to have this -ve cases also added to testWriteEnvExport itself.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thank you Sunil G for review.
          Will update patch after correcting checkstyle and adding one variable not available in white list but available in env

          Show
          bibinchundatt Bibin A Chundatt added a comment - Thank you Sunil G for review. Will update patch after correcting checkstyle and adding one variable not available in white list but available in env
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 46s trunk passed
          +1 compile 0m 27s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 41s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 24s the patch passed
          +1 javac 0m 24s the patch passed
          +1 checkstyle 0m 14s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 46s the patch passed
          +1 javadoc 0m 15s the patch passed
          +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          27m 11s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5877
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843435/YARN-5877.0006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 40746e4fe737 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 36947f7
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14330/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14330/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 41s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 27m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5877 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843435/YARN-5877.0006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 40746e4fe737 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 36947f7 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14330/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14330/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Sunil G
          Please do review.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Sunil G Please do review.
          Hide
          sunilg Sunil G added a comment -

          Thanks Bibin A Chundatt

          I tested this patch locally with LCE as well as with Regular Executor. Bibin A Chundatt also has tested in docker too.
          Patch generally looks fine for me +1. MAPREDUCE-6704 also seems working.

          Daniel Templeton, any comments? I could commit the patch later today if there are no objections.

          Show
          sunilg Sunil G added a comment - Thanks Bibin A Chundatt I tested this patch locally with LCE as well as with Regular Executor. Bibin A Chundatt also has tested in docker too. Patch generally looks fine for me +1. MAPREDUCE-6704 also seems working. Daniel Templeton , any comments? I could commit the patch later today if there are no objections.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching nm logs of test.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching nm logs of test.
          Hide
          sunilg Sunil G added a comment -

          Thanks Bibin A Chundatt for the logs. Looks fine for me.

          Newly added test case is also covering various cases as expected. I will commit the patch later today.

          Show
          sunilg Sunil G added a comment - Thanks Bibin A Chundatt for the logs. Looks fine for me. Newly added test case is also covering various cases as expected. I will commit the patch later today.
          Hide
          sunilg Sunil G added a comment -

          Thanks Bibin A Chundatt for the contribution and Varun Saxena Daniel Templeton Varun Vasudev for additional review. Committed to trunk!

          Show
          sunilg Sunil G added a comment - Thanks Bibin A Chundatt for the contribution and Varun Saxena Daniel Templeton Varun Vasudev for additional review. Committed to trunk!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11014 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11014/)
          YARN-5877. Allow all env's from yarn.nodemanager.env-whitelist to get (sunilg: rev 575773a3570b85293fdf7b8aeb8467580ec7f896)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11014 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11014/ ) YARN-5877 . Allow all env's from yarn.nodemanager.env-whitelist to get (sunilg: rev 575773a3570b85293fdf7b8aeb8467580ec7f896) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          Hide
          sunilg Sunil G added a comment -

          This patch will not be backported to branch-2 as environment variable inheritance is not broken in branch-2.

          Show
          sunilg Sunil G added a comment - This patch will not be backported to branch-2 as environment variable inheritance is not broken in branch-2.

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development