[RATIS-1483] Failed to load the required native library when starting RaftServer on aarch64 platform

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Component/s: thirdparty
• Environment: Running Platform: aarch64
• Assignee: Liu Jinghao
• Reporter: xuting
• Created: 06/Jan/22 12:54
• Updated: 09/Jan/22 11:51
• Resolved: 09/Jan/22 10:02

Description

Hello! When I starting RaftServer on aarch64 platform, the log reports the following errors.

2022-01-06 19:28:13.703 INFO  [raft-listener-thread-1] org.apache.ratis.server.RaftServer - raft.rpc.type = GRPC (default)
2022-01-06 19:28:13.836 INFO  [raft-listener-thread-1] org.apache.ratis.grpc.GrpcFactory - PERFORMANCE WARNING: useCacheForAllThreads is true that may cause Netty to create a lot garbage objects and, as a result, trigger GC.[N]     It is recommended to disable useCacheForAllThreads by setting -Dorg.apache.ratis.thirdparty.io.netty.allocator.useCacheForAllThreads=false in command line.
2022-01-06 19:28:13.849 INFO  [raft-listener-thread-1] org.apache.ratis.grpc.GrpcConfigKeys - raft.grpc.admin*****#*#*****
2022-01-06 19:28:13.852 INFO  [raft-listener-thread-1] org.apache.ratis.grpc.GrpcConfigKeys - raft.grpc.server.port = 22813 (custom)
2022-01-06 19:28:13.853 INFO  [raft-listener-thread-1] org.apache.ratis.grpc.GrpcConfigKeys - raft.grpc.client.port = -1 (default)
2022-01-06 19:28:13.853 INFO  [raft-listener-thread-1] org.apache.ratis.grpc.GrpcConfigKeys - raft.grpc.server.port = 22813 (custom)
2022-01-06 19:28:13.854 INFO  [raft-listener-thread-1] org.apache.ratis.grpc.GrpcConfigKeys - raft.grpc.server.port = 22813 (custom)
2022-01-06 19:28:13.855 INFO  [raft-listener-thread-1] org.apache.ratis.grpc.server.GrpcService - raft.grpc.message.size.max = 64MB (=67108864) (default)
2022-01-06 19:28:13.860 INFO  [raft-listener-thread-1] org.apache.ratis.grpc.server.GrpcService - raft.grpc.flow.control.window = 1MB (=1048576) (default)
2022-01-06 19:28:14.107 WARN  [raft-listener-thread-1] org.apache.ratis.server.RaftServer - FAILED "new RaftServerProxy", attempt #1/5: java.lang.UnsatisfiedLinkError: failed to load the required native library, sleep 500ms and then retry.
java.lang.UnsatisfiedLinkError: failed to load the required native library
        at org.apache.ratis.thirdparty.io.netty.handler.ssl.OpenSsl.ensureAvailability(OpenSsl.java:483) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.thirdparty.io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContext.java:207) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.thirdparty.io.netty.handler.ssl.OpenSslContext.<init>(OpenSslContext.java:45) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.thirdparty.io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:349) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.thirdparty.io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:336) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.thirdparty.io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:473) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.thirdparty.io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:609) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.grpc.server.GrpcService.startBuildingNettyServer(GrpcService.java:232) ~[ratis-grpc-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.grpc.server.GrpcService.<init>(GrpcService.java:157) ~[ratis-grpc-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.grpc.server.GrpcService.<init>(GrpcService.java:120) ~[ratis-grpc-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.grpc.server.GrpcService.<init>(GrpcService.java:52) ~[ratis-grpc-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.grpc.server.GrpcService$Builder.build(GrpcService.java:73) ~[ratis-grpc-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.grpc.GrpcFactory.newRaftServerRpc(GrpcFactory.java:121) ~[ratis-grpc-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.grpc.GrpcFactory.newRaftServerRpc(GrpcFactory.java:39) ~[ratis-grpc-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.server.impl.RaftServerProxy.<init>(RaftServerProxy.java:201) ~[ratis-server-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.server.impl.ServerImplUtils.lambda$newRaftServer$0(ServerImplUtils.java:63) [ratis-server-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.util.JavaUtils.attempt(JavaUtils.java:166) [ratis-common-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.util.JavaUtils.attemptRepeatedly(JavaUtils.java:152) [ratis-common-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:62) [ratis-server-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:50) [ratis-server-2.2.0.jar!/:2.2.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312]
        at org.apache.ratis.server.RaftServer$Builder.newRaftServer(RaftServer.java:173) [ratis-server-api-2.2.0.jar!/:2.2.0]
        at org.apache.ratis.server.RaftServer$Builder.build(RaftServer.java:190) [ratis-server-api-2.2.0.jar!/:2.2.0]
        at com.huawei.clouddr.drservice.raft.NodeServer.buildRaftServer(NodeServer.java:122) [clouddr-drservice-raft-6.1.3.0-SNAPSHOT.jar!/:na]
        at com.huawei.clouddr.drservice.raft.NodeServer.startRaftServer(NodeServer.java:88) [clouddr-drservice-raft-6.1.3.0-SNAPSHOT.jar!/:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_312]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_312]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312]
Caused by: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_aarch_64_fedora, netty_tcnative_linux_aarch_64, netty_tcnative_aarch_64, netty_tcnative]
        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:104) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.thirdparty.io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:592) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.thirdparty.io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:136) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.thirdparty.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:168) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
        at org.apache.ratis.grpc.server.GrpcService.startBuildingNettyServer(GrpcService.java:230) ~[ratis-grpc-2.2.0.jar!/:2.2.0]
        ... 25 common frames omitted
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: org_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64_fedora
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:221) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
                ... 29 common frames omitted
        Caused by: java.io.FileNotFoundException: META-INF/native/liborg_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64_fedora.so
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:170)
                ... 30 common frames omitted
                Suppressed: java.lang.UnsatisfiedLinkError: no org_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64_fedora in java.library.path
                        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
                        at java.lang.Runtime.loadLibrary0(Runtime.java:871)
                        at java.lang.System.loadLibrary(System.java:1124)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:344)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 30 common frames omitted
                        Suppressed: java.lang.UnsatisfiedLinkError: no org_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64_fedora in java.library.path
                                at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
                                at java.lang.Runtime.loadLibrary0(Runtime.java:871)
                                at java.lang.System.loadLibrary(System.java:1124)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                at java.lang.reflect.Method.invoke(Method.java:498)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:378)
                                at java.security.AccessController.doPrivileged(Native Method)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:370)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:336)
                                ... 31 common frames omitted
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: org_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:221) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
                ... 29 common frames omitted
        Caused by: java.io.FileNotFoundException: META-INF/native/liborg_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64.so
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:170)
                ... 30 common frames omitted
                Suppressed: java.lang.UnsatisfiedLinkError: no org_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64 in java.library.path
                        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
                        at java.lang.Runtime.loadLibrary0(Runtime.java:871)
                        at java.lang.System.loadLibrary(System.java:1124)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:344)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 30 common frames omitted
                        Suppressed: java.lang.UnsatisfiedLinkError: no org_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64 in java.library.path
                                at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
                                at java.lang.Runtime.loadLibrary0(Runtime.java:871)
                                at java.lang.System.loadLibrary(System.java:1124)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                at java.lang.reflect.Method.invoke(Method.java:498)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:378)
                                at java.security.AccessController.doPrivileged(Native Method)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:370)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:336)
                                ... 31 common frames omitted
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: org_apache_ratis_thirdparty_netty_tcnative_aarch_64
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:221) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
                ... 29 common frames omitted
        Caused by: java.io.FileNotFoundException: META-INF/native/liborg_apache_ratis_thirdparty_netty_tcnative_aarch_64.so
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:170)
                ... 30 common frames omitted
                Suppressed: java.lang.UnsatisfiedLinkError: no org_apache_ratis_thirdparty_netty_tcnative_aarch_64 in java.library.path
                        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
                        at java.lang.Runtime.loadLibrary0(Runtime.java:871)
                        at java.lang.System.loadLibrary(System.java:1124)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:344)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 30 common frames omitted
                        Suppressed: java.lang.UnsatisfiedLinkError: no org_apache_ratis_thirdparty_netty_tcnative_aarch_64 in java.library.path
                                at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
                                at java.lang.Runtime.loadLibrary0(Runtime.java:871)
                                at java.lang.System.loadLibrary(System.java:1124)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                at java.lang.reflect.Method.invoke(Method.java:498)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:378)
                                at java.security.AccessController.doPrivileged(Native Method)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:370)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:336)
                                ... 31 common frames omitted
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: org_apache_ratis_thirdparty_netty_tcnative
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:221) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96) ~[ratis-thirdparty-misc-0.7.0.jar!/:0.7.0]
                ... 29 common frames omitted
        Caused by: java.io.FileNotFoundException: META-INF/native/liborg_apache_ratis_thirdparty_netty_tcnative.so
                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:170)
                ... 30 common frames omitted
                Suppressed: java.lang.UnsatisfiedLinkError: no org_apache_ratis_thirdparty_netty_tcnative in java.library.path
                        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
                        at java.lang.Runtime.loadLibrary0(Runtime.java:871)
                        at java.lang.System.loadLibrary(System.java:1124)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:344)
                        at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 30 common frames omitted
                        Suppressed: java.lang.UnsatisfiedLinkError: no org_apache_ratis_thirdparty_netty_tcnative in java.library.path
                                at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
                                at java.lang.Runtime.loadLibrary0(Runtime.java:871)
                                at java.lang.System.loadLibrary(System.java:1124)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                at java.lang.reflect.Method.invoke(Method.java:498)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:378)
                                at java.security.AccessController.doPrivileged(Native Method)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:370)
                                at org.apache.ratis.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:336)
                                ... 31 common frames omitted
 

And here is my pom.xml:

<dependency>
    <groupId>org.apache.ratis</groupId>
    <artifactId>ratis-grpc</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.ratis</groupId>
    <artifactId>ratis-server</artifactId>
    <version>2.2.0</version>
</dependency> 

And here is the auto-downloaded ratis-thirdparty-misc-0.7.0.jar which lacks liborg_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64_fedora.so, liborg_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64.so, liborg_apache_ratis_thirdparty_netty_tcnative_aarch_64.so and  liborg_apache_ratis_thirdparty_netty_tcnative.so.

Is there any solution?

Thanks you for your answers!

 

Issue Links

• duplicates: RATIS-1443 Do not provide non-shaded Netty native libraries
• links to: RATIS-1443 Shade all Netty Native libraries