Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17891

lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g. hadoop-client-api, if we don't exclude lz4 dependency, the downstream will still see the exception even they include lz4 dependency.

      [info]   Cause: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
      [info]   at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
      [info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
      [info]   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
      [info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      [info]   at org.apache.hadoop.io.compress.lz4.Lz4Compressor.<init>(Lz4Compressor.java:66)
      [info]   at org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
      [info]   at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
      [info]   at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
       

      Currently snappy-java is included and relocated in Hadoop shaded client libraries. But as it includes native methods, it should not be relocated too due to JNI method resolution. The downstream will see the exception:

      [info]   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
      [info]   at org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native Method)                                                                                                 
      [info]   at org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)                                                                                                        
      [info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
      [info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
      
      

      Attachments

        1. HADOOP-17891-Addendum-01.patch
          0.9 kB
          Ayush Saxena

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            viirya L. C. Hsieh
            viirya L. C. Hsieh
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 17h 40m
                17h 40m

                Slack

                  Issue deployment