[SPARK-36681] Fail to load Snappy codec

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: Spark Core
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 07/Sep/21 07:41
• Updated: 23/Jun/23 09:47
• Resolved: 09/Mar/22 17:08

Description

snappy-java as a native library should not be relocated in Hadoop shaded client libraries. Currently we use Hadoop shaded client libraries in Spark. If trying to use SnappyCodec to write sequence file, we will encounter the following error:

[info]   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native Method)                                                                                                 
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)                                                                                                        
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
[info]   at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
[info]   at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
[info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.writeBuffer(SequenceFile.java:1589) 
[info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.sync(SequenceFile.java:1605)
[info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1629) 

Issue Links

• is caused by: HADOOP-17891 lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
• relates to: HADOOP-17125 Using snappy-java in SnappyCodec
• links to: [Github] Pull Request #35784 (viirya)
• links to: [Github] Pull Request #36136 (HyukjinKwon)
• links to: [Github] Pull Request #36136 (HyukjinKwon)
• links to: [Github] Pull Request #36687 (peter-toth)