Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12094

Log torrent when none isa-l EC is used.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: 3.0.0-beta1
    • Component/s: erasure-coding
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Flags:
      Patch

      Description

      My hadoop is built without isa-l support, after the EC policy is enabled, whenever I get/put directory which contains many files, the log of warnings(see below) spam on the screen!
      This is unfriendly and depress the performance. Since we come to the beta version now,
      these logs should be deprecated and a one-time warning log instead of exception may be much better.

      2017-07-06 15:42:41,398 WARN erasurecode.CodecUtil: Failed to create raw erasure encoder xor_native, fallback to next codec if possible
      java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawEncoder
      at org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory.createEncoder(NativeXORRawErasureCoderFactory.java:35)
      at org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:177)
      at org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:129)
      at org.apache.hadoop.hdfs.DFSStripedOutputStream.<init>(DFSStripedOutputStream.java:302)
      at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:309)
      at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1216)
      at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
      at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1133)
      ...
      at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)

      Caused by: java.lang.RuntimeException: libhadoop was built without ISA-L support
      at org.apache.hadoop.io.erasurecode.ErasureCodeNative.checkNativeCodeLoaded(ErasureCodeNative.java:69)
      at org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.<clinit>(NativeXORRawDecoder.java:33)
      ... 25 more

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                GeLiXin LiXin Ge
                Reporter:
                GeLiXin LiXin Ge
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: