Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5451

MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1-beta, 3.0.0-alpha1
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Windows

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In order to set the path for loading native libraries, MR relies on the default value of the mapreduce.admin.user.env configuration setting the LD_LIBRARY_PATH environment entry. There are two problems with this setting in Windows:
      a) LD_LIBRARY_PATH doesn't mean anything in Windows.
      b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%.

      The default value here should be platform-dependent (use the PATH variable in Windows instead of LD_LIBRARY_PATH), or we should rely on another mechanism. The net effect is that in Windows unless this configuration is over-ridden MR jobs fail with this error:

      2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
      	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
      	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393)
      	at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928)
      	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177)
      	at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164)
      	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98)
      	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288)
      	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431)
      	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
      	at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235)
      	at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
      

        Attachments

        1. MAPREDUCE-5451.patch
          5 kB
          Yingda Chen
        2. MAPREDUCE-5451-2.patch
          4 kB
          Yingda Chen
        3. MAPREDUCE-5451-3.patch
          4 kB
          Yingda Chen
        4. MAPREDUCE-5451-4.patch
          5 kB
          Chris Nauroth
        5. MAPREDUCE-5451-5.patch
          8 kB
          Yingda Chen
        6. MAPREDUCE-5451-6.patch
          8 kB
          Yingda Chen
        7. MAPREDUCE-5451-7.patch
          8 kB
          Yingda Chen
        8. YARN-729.patch
          5 kB
          Yingda Chen

          Issue Links

            Activity

              People

              • Assignee:
                yingdachen Yingda Chen
                Reporter:
                mostafae Mostafa Elhemali
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: