Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36681

Fail to load Snappy codec

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.0
    • None
    • Spark Core
    • None

    Description

      snappy-java as a native library should not be relocated in Hadoop shaded client libraries. Currently we use Hadoop shaded client libraries in Spark. If trying to use SnappyCodec to write sequence file, we will encounter the following error:

      [info]   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
      [info]   at org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native Method)                                                                                                 
      [info]   at org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)                                                                                                        
      [info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
      [info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
      [info]   at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
      [info]   at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
      [info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.writeBuffer(SequenceFile.java:1589) 
      [info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.sync(SequenceFile.java:1605)
      [info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1629) 
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              viirya L. C. Hsieh
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: