Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19387

HBase-spark snappy.SnappyError on Arm64

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed

      Description

      When running the hbase-spark Unit tests on Arm64, the failures are shown as follows:

      scalatest-maven-plugin:1.0:test (test) @ hbase-spark ---
      Discovery starting.
      Discovery completed in 2 seconds, 837 milliseconds.
      Run starting. Expected test count is: 79
      HBaseDStreamFunctionsSuite:
      Formatting using clusterid: testClusterID
      - bulkput to test HBase client *** FAILED ***
        java.lang.reflect.InvocationTargetException:
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
        at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
        at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
        at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
        ...
        Cause: java.lang.IllegalArgumentException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=aarch64
        at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
        at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
        at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
        ...
        Cause: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=aarch64
        at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:331)
        at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:171)
        at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:152)
        at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
        at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
        ...
      Formatting using clusterid: testClusterID
      PartitionFilterSuite:
      *** RUN ABORTED ***
        java.lang.reflect.InvocationTargetException:
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
        at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
        at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
        at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
        ...
        Cause: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
        at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
        at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
        at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
        ...
        Cause: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
        at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
        at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
        at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
      

      Root Cause:
      The spark version in hbase is 1.6.0.
      And the snappy version based on hbase-1.6.0 do not support the Arm64.
      Fix it by upgrading snappy-java to 1.1.4 in hbase-spark.

        Attachments

        1. HBASE-19387.patch
          1 kB
          Yuqi Gu

          Activity

            People

            • Assignee:
              yqGu Yuqi Gu
              Reporter:
              yqGu Yuqi Gu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: