Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6577

MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: mr-am
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:

      2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      

      As a result, any code that needs the hadoop native library in the MR AM will fail. For example, an uber-AM with lz4 compression for the mapper task will fail:

      2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local (uberized) 'child' : java.lang.RuntimeException: native lz4 library not available
      	at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
      	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
      	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
      	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
      	at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
      	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
      	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
      	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
      	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
      	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
      	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        1. MAPREDUCE-6577.04.patch
          13 kB
          Sangjin Lee
        2. MAPREDUCE-6577.03.patch
          13 kB
          Sangjin Lee
        3. MAPREDUCE-6577.02.patch
          9 kB
          Sangjin Lee
        4. MAPREDUCE-6577.01.patch
          9 kB
          Sangjin Lee

          Issue Links

            Activity

              People

              • Assignee:
                sjlee0 Sangjin Lee
                Reporter:
                sjlee0 Sangjin Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: