[HBASE-19387] HBase-spark snappy.SnappyError on Arm64

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 3.0.0-alpha-1, connector-1.0.0
• Component/s: hbase-connectors, spark, test
• Hadoop Flags: Reviewed
• Assignee: Yuqi Gu
• Reporter: Yuqi Gu
• Created: 30/Nov/17 07:20
• Updated: 26/Apr/19 09:45
• Resolved: 08/Dec/17 17:23

Description

When running the hbase-spark Unit tests on Arm64, the failures are shown as follows:

scalatest-maven-plugin:1.0:test (test) @ hbase-spark ---
Discovery starting.
Discovery completed in 2 seconds, 837 milliseconds.
Run starting. Expected test count is: 79
HBaseDStreamFunctionsSuite:
Formatting using clusterid: testClusterID
- bulkput to test HBase client *** FAILED ***
  java.lang.reflect.InvocationTargetException:
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
  at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
  at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
  at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
  at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
  ...
  Cause: java.lang.IllegalArgumentException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=aarch64
  at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
  at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
  at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
  at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
  ...
  Cause: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=aarch64
  at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:331)
  at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:171)
  at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:152)
  at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
  at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
  ...
Formatting using clusterid: testClusterID
PartitionFilterSuite:
*** RUN ABORTED ***
  java.lang.reflect.InvocationTargetException:
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
  at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
  at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
  at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
  at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
  ...
  Cause: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
  at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
  at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
  at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
  at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
  ...
  Cause: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
  at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72)
  at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65)
  at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
  at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
  at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)

Root Cause:
The spark version in hbase is 1.6.0.
And the snappy version based on hbase-1.6.0 do not support the Arm64.
Fix it by upgrading snappy-java to 1.1.4 in hbase-spark.