[MAPREDUCE-6590] Update MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV value for loading correct navite lib when using ubertask feature

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.7.1
• Component/s: mr-am
• Assignee: NING DING
• Reporter: NING DING
• Created: 27/Dec/15 01:24
• Updated: 05/Jan/16 21:55
• Resolved: 05/Jan/16 21:55

Description

By default, the MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library, then you can find the error message as below.

org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable.

I found the patch in https://issues.apache.org/jira/browse/MAPREDUCE-5799 update YARNRunner code and use MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV as the default value. The code is as below.

YARNRunner.java

    MRApps.setEnvFromInputString(environment, 
        conf.get(MRJobConfig.MR_AM_ADMIN_USER_ENV,
            MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV), conf);

In fact the parameters yarn.app.mapreduce.am.env and yarn.app.mapreduce.am.admin.user.env are resolved in YARNRunner running on hadoop client host.
But their values, e.g. LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, will be used on hadoop cluster hosts.
If the hadoop client host installs hadoop on a different path from that on hadoop cluster hosts,
the $HADOOP_COMMON_HOME/lib/native will be resolved to a path which is not existed on hadoop cluster hosts.
Then MRAppMaster running on hadoop cluster host cannot load native lib.
If don't use MAPREDUCE-5799 patch, I set my hadoop client's mapred-site.xml with the following content.

<property>
        <name>yarn.app.mapreduce.am.admin.user.env</name>
        <value>LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native</value>        
</property>

In this way, the YARNRunner will put LD_LIBRARY_PATH=HADOOP_COMMON_HOME/lib/native into environment.
The method ContainerLaunch.expandEnvironment running in NodeManager can translate LD_LIBRARY_PATH=HADOOP_COMMON_HOME/lib/native
to LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
The host running NodeManager can find its $HADOOP_COMMON_HOME/lib/native path.

I suggest MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV should be defined as below.

MRJobConfig.java

public final String DEFAULT_MAPRED_ADMIN_USER_ENV = 
      Shell.WINDOWS ? 
          "PATH={{PATH}};{{HADOOP_COMMON_HOME}}\\bin":
          "LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native";

Please see my patch.

Issue Links

• duplicates: MAPREDUCE-6577 MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set