Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4072

User set java.library.path seems to overwrite default creating problems native lib loading

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      -Djava.library.path in mapred.child.java.opts can cause issues with native libraries. LD_LIBRARY_PATH through mapred.child.env should be used instead.

      Description

      This was found by Peeyush Bishnoi.

      While running a distributed cache example with Hadoop-0.23,
      tasks are failing as follows:
      ------------------------------------------------------------------------------------------------------------

      Exception from container-launch:
      org.apache.hadoop.util.Shell$ExitCodeException: at
      org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
      org.apache.hadoop.util.Shell.run(Shell.java:188) at
      org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
      org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
      at
      org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
      at
      org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
      java.util.concurrent.FutureTask.run(FutureTask.java:138) at
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
      is <user>
      ------------------------------------------------------------------------------------------------------------

      Same Pig script and command work successfully on 0.20

      See this in the stderr:

      Exception in thread "main" java.lang.ExceptionInInitializerError
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:247)
      at
      org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
      at
      org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
      at org.apache.hadoop.security.Groups.(Groups.java:54)
      at
      org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
      at
      org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
      at
      org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
      at
      org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
      Caused by: java.lang.RuntimeException: Bailing out since native library
      couldn't be loaded
      at
      org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
      ... 12 more

      Pig command:
      $ pig -Dmapred.job.queue.name=<queue> -Dmapred.cache.archives=<archives> -Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib
      -Dip2geo.preLoadLibraries=<some other libs>" -Djava.io.tmpdir=/grid/0/tmp -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Anupam Seth
              Reporter:
              Anupam Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development