[ARROW-17267] Failure loading arrow_dataset_jni library on Mac M1

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 8.0.0
• Component/s: Java
• Environment: Mac M1, 16 GB memory
• Flags: Important
• External issue URL: https://github.com/apache/arrow/issues/32547
• Assignee: Rok Mihevc
• Reporter: Alexander Cai
• Created: 30/Jul/22 15:52
• Updated: 11/Jan/23 11:49
• Resolved: 13/Oct/22 17:10

Description

On a Mac M1, using Java 11 and Maven, I was initially trying to load a Parquet file using FileSystemDatasetFactory. However, when calling NativeMemoryPool.getDefault(), it throws an UnsatisfiedLinkError:

java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/c9/h03f8v4s037gvwlfgs98b_y00000gp/T/jnilib-18262759266549290103.tmp
    at java.lang.ClassLoader.loadLibrary (ClassLoader.java:2633)
    at java.lang.Runtime.load0 (Runtime.java:768)
    at java.lang.System.load (System.java:1837)
    at org.apache.arrow.dataset.jni.JniLoader.load (JniLoader.java:88)
    at org.apache.arrow.dataset.jni.JniLoader.loadRemaining (JniLoader.java:73)
    at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded (JniLoader.java:60)
    at org.apache.arrow.dataset.jni.NativeMemoryPool.<clinit> (NativeMemoryPool.java:27)
    at com.example.App.main (App.java:8)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:829)

I've created a minimum reproducible example here, where the only line of code is

NativeMemoryPool.getDefault()

. At the call to

org.apache.arrow.dataset.jni.JniLoader.load

in the stack trace above, "arrow_dataset_jni" is passed as the name, and "libarrow_dataset_jni.dylib" is the library name, which it successfully locates.

However, this works fine in Java 8. Running the same code in an EC2 container on Java 11 also works fine, so I'm fairly certain this is an M1 issue.

 

Issue Links

• is duplicated by: ARROW-16608 [Gandiva][Java] Unsatisfied Link Error on M1 Mac when using mavencentral artifacts