[HADOOP-17891] lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.4.0, 3.3.2
• Component/s: build
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.2
• Hadoop Flags: Reviewed
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 04/Sep/21 21:33
• Updated: 11/Feb/24 23:04
• Resolved: 14/Sep/21 18:17

Description

lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g. hadoop-client-api, if we don't exclude lz4 dependency, the downstream will still see the exception even they include lz4 dependency.

[info]   Cause: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
[info]   at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[info]   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[info]   at org.apache.hadoop.io.compress.lz4.Lz4Compressor.<init>(Lz4Compressor.java:66)
[info]   at org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
[info]   at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
[info]   at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
 

Currently snappy-java is included and relocated in Hadoop shaded client libraries. But as it includes native methods, it should not be relocated too due to JNI method resolution. The downstream will see the exception:

[info]   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native Method)                                                                                                 
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)                                                                                                        
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)

Issue Links

• causes: SPARK-36681 Fail to load Snappy codec
• is duplicated by: HADOOP-17916 Fix compilation error of ITUseHadoopCodecs with -DskipShade
• is related to: HADOOP-17125 Using snappy-java in SnappyCodec
• is related to: HADOOP-17292 Using lz4-java in Lz4Codec
• relates to: SPARK-36669 Fail to load Lz4 codec
• links to: GitHub Pull Request #3385
• links to: GitHub Pull Request #3441