Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4072

User set java.library.path seems to overwrite default creating problems native lib loading

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      -Djava.library.path in mapred.child.java.opts can cause issues with native libraries. LD_LIBRARY_PATH through mapred.child.env should be used instead.

      Description

      This was found by Peeyush Bishnoi.

      While running a distributed cache example with Hadoop-0.23,
      tasks are failing as follows:
      ------------------------------------------------------------------------------------------------------------

      Exception from container-launch:
      org.apache.hadoop.util.Shell$ExitCodeException: at
      org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
      org.apache.hadoop.util.Shell.run(Shell.java:188) at
      org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
      org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
      at
      org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
      at
      org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
      java.util.concurrent.FutureTask.run(FutureTask.java:138) at
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
      is <user>
      ------------------------------------------------------------------------------------------------------------

      Same Pig script and command work successfully on 0.20

      See this in the stderr:

      Exception in thread "main" java.lang.ExceptionInInitializerError
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:247)
      at
      org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
      at
      org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
      at org.apache.hadoop.security.Groups.(Groups.java:54)
      at
      org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
      at
      org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
      at
      org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
      at
      org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
      Caused by: java.lang.RuntimeException: Bailing out since native library
      couldn't be loaded
      at
      org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
      ... 12 more

      Pig command:
      $ pig -Dmapred.job.queue.name=<queue> -Dmapred.cache.archives=<archives> -Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib
      -Dip2geo.preLoadLibraries=<some other libs>" -Djava.io.tmpdir=/grid/0/tmp -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1040 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/)
          MAPREDUCE-4072. User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1040 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/ ) MAPREDUCE-4072 . User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1005 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/)
          MAPREDUCE-4072. User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1005 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/ ) MAPREDUCE-4072 . User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #218 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/218/)
          svn merge -c 1309077 from trunk. FIXES MAPREDUCE-4072. User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309080)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309080
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #218 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/218/ ) svn merge -c 1309077 from trunk. FIXES MAPREDUCE-4072 . User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309080) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309080 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1994 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1994/)
          MAPREDUCE-4072. User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1994 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1994/ ) MAPREDUCE-4072 . User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1981 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1981/)
          MAPREDUCE-4072. User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1981 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1981/ ) MAPREDUCE-4072 . User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2056 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2056/)
          MAPREDUCE-4072. User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2056 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2056/ ) MAPREDUCE-4072 . User set java.library.path seems to overwrite default creating problems native lib loading (Anupam Seth via bobby) (Revision 1309077) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309077 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Hide
          Robert Joseph Evans added a comment -

          Thanks Anupam. +1 on the documentation. I have checked this into trunk, branch-2, and branch-0.23

          Show
          Robert Joseph Evans added a comment - Thanks Anupam. +1 on the documentation. I have checked this into trunk, branch-2, and branch-0.23
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521161/MAPREDUCE-4072-branch-23_documentation.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2134//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2134//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521161/MAPREDUCE-4072-branch-23_documentation.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2134//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2134//console This message is automatically generated.
          Hide
          Anupam Seth added a comment -

          Thanks bobby! I have updated the patch to include a pointer to conifg settings for the child JVM.

          Show
          Anupam Seth added a comment - Thanks bobby! I have updated the patch to include a pointer to conifg settings for the child JVM.
          Hide
          Robert Joseph Evans added a comment -

          I agree that doc only is the preferable solution. I like the documentation. I think it would be good to have in the mapred-default.xml mapred.child.java.opts when it talks about LD_LIBRARY_PATH have it point to mapred.child.env.

          Show
          Robert Joseph Evans added a comment - I agree that doc only is the preferable solution. I like the documentation. I think it would be good to have in the mapred-default.xml mapred.child.java.opts when it talks about LD_LIBRARY_PATH have it point to mapred.child.env.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521053/MAPREDUCE-4072-branch-23_documentation.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
          org.apache.hadoop.mapreduce.v2.TestRMNMInfo
          org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
          org.apache.hadoop.mapred.TestLazyOutput
          org.apache.hadoop.mapreduce.TestChild
          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapred.TestMiniMRBringup
          org.apache.hadoop.mapred.TestMiniMRClientCluster
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
          org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
          org.apache.hadoop.mapreduce.v2.TestNonExistentJob
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapred.TestJobCounters
          org.apache.hadoop.mapreduce.v2.TestUberAM
          org.apache.hadoop.conf.TestNoDefaultsJobConf
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.mapreduce.v2.TestMRJobs
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapred.TestJobCleanup
          org.apache.hadoop.mapreduce.TestMapReduceLazyOutput

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2131//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2131//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521053/MAPREDUCE-4072-branch-23_documentation.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2131//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2131//console This message is automatically generated.
          Hide
          Anupam Seth added a comment -

          One reason we might want to go for a doc change instead of stripping out and copying the java.library.path set by the user to LD_LIBRARY_PATH is because it is cleaner to set it right in the first place and we likely don't want the overhead of supporting such in the long run.

          Quoting Kihwal to help clarify:
          "If only LD_LIBRARY_PATH is specified, the JVM (at least's Oracle's) resorts to the dynamic linker to search for the library. If java.library.path is set, the JVM will only look at the paths in java.library.path. Since the search capability of the linker is superior, we prefer using the former method whenever possible. The JVM resource loading stops searching when it locates one match, whether the match is for the correct architecture or not. This can be troublesome when supporting multiple architectures, which is a goal of yarn.

          LD_LIBRARY_PATH is also needed if there is a chain of dependencies on native libraries from a JNI library and those are placed in a non-standard directory(i.e. non-system dirs and not defined in /etc/ld.so.conf.d/*)."

          Show
          Anupam Seth added a comment - One reason we might want to go for a doc change instead of stripping out and copying the java.library.path set by the user to LD_LIBRARY_PATH is because it is cleaner to set it right in the first place and we likely don't want the overhead of supporting such in the long run. Quoting Kihwal to help clarify: "If only LD_LIBRARY_PATH is specified, the JVM (at least's Oracle's) resorts to the dynamic linker to search for the library. If java.library.path is set, the JVM will only look at the paths in java.library.path. Since the search capability of the linker is superior, we prefer using the former method whenever possible. The JVM resource loading stops searching when it locates one match, whether the match is for the correct architecture or not. This can be troublesome when supporting multiple architectures, which is a goal of yarn. LD_LIBRARY_PATH is also needed if there is a chain of dependencies on native libraries from a JNI library and those are placed in a non-standard directory(i.e. non-system dirs and not defined in /etc/ld.so.conf.d/*)."
          Hide
          Anupam Seth added a comment -

          Thanks Bobby and Kihwal. I apologize for my lack of understanding.

          I am removing the code change and uploading a doc only patch to incorporate your feedback.

          Show
          Anupam Seth added a comment - Thanks Bobby and Kihwal. I apologize for my lack of understanding. I am removing the code change and uploading a doc only patch to incorporate your feedback.
          Hide
          Kihwal Lee added a comment -

          I don't think that this is the correct place to make these changes.

          You are right. I think there used to be one in launchContainer() but that must have already been removed. Since the resource localizer is a part of YARN, we should leave it as is.

          Show
          Kihwal Lee added a comment - I don't think that this is the correct place to make these changes. You are right. I think there used to be one in launchContainer() but that must have already been removed. Since the resource localizer is a part of YARN, we should leave it as is.
          Hide
          Robert Joseph Evans added a comment -

          I wanted to add in that I am OK with removing the lines in the patch, so long as you have tested this on a cluster with the Linux container executor enabled and also the native libraries to verify that those lines are not needed.

          Show
          Robert Joseph Evans added a comment - I wanted to add in that I am OK with removing the lines in the patch, so long as you have tested this on a cluster with the Linux container executor enabled and also the native libraries to verify that those lines are not needed.
          Hide
          Robert Joseph Evans added a comment -

          I don't think that this is the correct place to make these changes. You are removing using java.library.path from the localizer command line. This is the part of the code that will download a file from HDFS to the local file system as a particular user. Is this the process that was causing the error? I don't see how because it has nothing to do with mapred.child.java.opts. mapred.child.java.opts is something that the MRAM will use when produing the container launch context that it sends to the NM.

          So we either need to update the AM strip out java.library.path from mapred.child.java.opts, and add it to LD_LIBRARY_PATH, or just put in a documentation change. In the mapred-default.xml under mapred.child.java.opts say that the usage of -Djava.library.path can cause your programs to no longer function and that these values should be set as part of LD_LIBRARY_PATH.

          In either case we probably also want to update hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm to have something in it about how setting -Djava.library.path on the command line while launching a container can cause native libraries used by hadoop to not be loaded correctly and can result in errors. It is better to just use LD_LIBRARY_PATH.

          Show
          Robert Joseph Evans added a comment - I don't think that this is the correct place to make these changes. You are removing using java.library.path from the localizer command line. This is the part of the code that will download a file from HDFS to the local file system as a particular user. Is this the process that was causing the error? I don't see how because it has nothing to do with mapred.child.java.opts. mapred.child.java.opts is something that the MRAM will use when produing the container launch context that it sends to the NM. So we either need to update the AM strip out java.library.path from mapred.child.java.opts, and add it to LD_LIBRARY_PATH, or just put in a documentation change. In the mapred-default.xml under mapred.child.java.opts say that the usage of -Djava.library.path can cause your programs to no longer function and that these values should be set as part of LD_LIBRARY_PATH. In either case we probably also want to update hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm to have something in it about how setting -Djava.library.path on the command line while launching a container can cause native libraries used by hadoop to not be loaded correctly and can result in errors. It is better to just use LD_LIBRARY_PATH.
          Hide
          Kihwal Lee added a comment -

          Users should not include -Djava.library.path in task jvm options. The code change looks good, but we need to make sure this is documented somewhere as an incompatible change.

          Show
          Kihwal Lee added a comment - Users should not include -Djava.library.path in task jvm options. The code change looks good, but we need to make sure this is documented somewhere as an incompatible change.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520642/MAPREDUCE-4072-branch-23.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2118//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2118//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520642/MAPREDUCE-4072-branch-23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2118//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2118//console This message is automatically generated.
          Hide
          Anupam Seth added a comment -

          Uploading patch to remove transmission of java.library.path to container.

          Show
          Anupam Seth added a comment - Uploading patch to remove transmission of java.library.path to container.
          Hide
          Arun C Murthy added a comment -

          We should avoid setting -Djava.library.path when launching container and instead set LD_LIBRARY_PATH correctly.

          +1!

          Show
          Arun C Murthy added a comment - We should avoid setting -Djava.library.path when launching container and instead set LD_LIBRARY_PATH correctly. +1!
          Hide
          Kihwal Lee added a comment -

          I tested a bit.

          If we are to rely on LD_LIBRARY_PATH, there should be NO -Djava.library.path. Once this property is set, the JVM only searches the paths in java.library.path for loading JNI libraries and ignores LD_LIBRARY_PATH. I think this was the design in 0.23. It means that if users supply -Djava.library.path in client map/red options, something can break.

          Using LD_LIBRARY_PATH is better since the JVM relies on the system function to locate the correct library, whereas specifying java.library.path causes the JVM's own resource loader to be in charge, which can be troublesome if the path contains a library with the same name but for different architecture.

          1. The bug of unconditionally specifying -Djava.library.path for child jvm needs to be fixed.
          2. Educate users that use of -Djava.library.path is no longer supported. They should specify LD_LIBRARY_PATH in the child map/red jvm env config. This is an incompatible change in 0.23.

          If we want to support java.library.path, we have to reintroduce the java.library.path munging code. This will make the system compatible, but has its own share of problems and not is against the goal of the YARN architecture.

          We should avoid setting -Djava.library.path when launching container and instead set LD_LIBRARY_PATH correctly.

          Show
          Kihwal Lee added a comment - I tested a bit. If we are to rely on LD_LIBRARY_PATH, there should be NO -Djava.library.path. Once this property is set, the JVM only searches the paths in java.library.path for loading JNI libraries and ignores LD_LIBRARY_PATH. I think this was the design in 0.23. It means that if users supply -Djava.library.path in client map/red options, something can break. Using LD_LIBRARY_PATH is better since the JVM relies on the system function to locate the correct library, whereas specifying java.library.path causes the JVM's own resource loader to be in charge, which can be troublesome if the path contains a library with the same name but for different architecture. 1. The bug of unconditionally specifying -Djava.library.path for child jvm needs to be fixed. 2. Educate users that use of -Djava.library.path is no longer supported. They should specify LD_LIBRARY_PATH in the child map/red jvm env config. This is an incompatible change in 0.23. If we want to support java.library.path, we have to reintroduce the java.library.path munging code. This will make the system compatible, but has its own share of problems and not is against the goal of the YARN architecture. We should avoid setting -Djava.library.path when launching container and instead set LD_LIBRARY_PATH correctly.
          Hide
          Anupam Seth added a comment -

          Looks like this was brought up earlier, but we actually need both java.library.path and LD_LIBRARY_PATH to be set in the container JVM to avoid the issue this described in this JIRA from happening.

          Show
          Anupam Seth added a comment - Looks like this was brought up earlier, but we actually need both java.library.path and LD_LIBRARY_PATH to be set in the container JVM to avoid the issue this described in this JIRA from happening.
          Hide
          Kihwal Lee added a comment -

          It look like MAPREDUCE-2880 got rid of merging of java.library.path and introduced setEnvFromInputString(). However this does not take care of the cases for java.library.path. When launching a container, "-Djava.library.path=" should contain both the system jni lib path (i.e. the location of libhadoop.so) and whatever user specified.

          Please note that setting LD_LIBRARY_PATH alone is not enough to load JNI libraries. A custom LD_LIBRARY_PATH needs to be used when the user supplies custom native libraries, which the JNI library depends on, but the base system does not provide.

          setEnvFromInputString() does path munging on LD_LIBRARY_PATH.

          Show
          Kihwal Lee added a comment - It look like MAPREDUCE-2880 got rid of merging of java.library.path and introduced setEnvFromInputString(). However this does not take care of the cases for java.library.path. When launching a container, "-Djava.library.path=" should contain both the system jni lib path (i.e. the location of libhadoop.so) and whatever user specified. Please note that setting LD_LIBRARY_PATH alone is not enough to load JNI libraries. A custom LD_LIBRARY_PATH needs to be used when the user supplies custom native libraries, which the JNI library depends on, but the base system does not provide. setEnvFromInputString() does path munging on LD_LIBRARY_PATH.

            People

            • Assignee:
              Anupam Seth
              Reporter:
              Anupam Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development