Affects Version/s: 2.7.1
Fix Version/s: None
By default, the MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library, then you can find the error message as below.
I found the patch in https://issues.apache.org/jira/browse/MAPREDUCE-5799 update YARNRunner code and use MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV as the default value. The code is as below.
In fact the parameters yarn.app.mapreduce.am.env and yarn.app.mapreduce.am.admin.user.env are resolved in YARNRunner running on hadoop client host.
But their values, e.g. LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, will be used on hadoop cluster hosts.
If the hadoop client host installs hadoop on a different path from that on hadoop cluster hosts,
the $HADOOP_COMMON_HOME/lib/native will be resolved to a path which is not existed on hadoop cluster hosts.
Then MRAppMaster running on hadoop cluster host cannot load native lib.
If don't use
MAPREDUCE-5799 patch, I set my hadoop client's mapred-site.xml with the following content.
In this way, the YARNRunner will put LD_LIBRARY_PATH=HADOOP_COMMON_HOME/lib/native into environment.
The method ContainerLaunch.expandEnvironment running in NodeManager can translate LD_LIBRARY_PATH=HADOOP_COMMON_HOME/lib/native
The host running NodeManager can find its $HADOOP_COMMON_HOME/lib/native path.
I suggest MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV should be defined as below.
Please see my patch.