[FLINK-24932] Frocksdb cannot run on Apple M1

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.17.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: Yun Tang
• Reporter: Yun Tang
• Created: 17/Nov/21 06:54
• Updated: 28/Jan/23 03:33
• Resolved: 28/Jan/23 03:33

Description

After we bump up RocksDB version to 6.20.3, we support to run RocksDB on linux arm cluster. However, according to the feedback from Robert, Apple M1 machines cannot run FRocksDB yet:

java.lang.Exception: Exception while creating StreamOperatorStateContext.
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-runtime-1.14.0.jar:1.14.0]
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-runtime-1.14.0.jar:1.14.0]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) ~[flink-runtime-1.14.0.jar:1.14.0]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-runtime-1.14.0.jar:1.14.0]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from any of the 1 provided restore options.
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	... 11 more
Caused by: java.io.IOException: Could not load the native RocksDB library
	at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	... 11 more
Caused by: java.lang.UnsatisfiedLinkError: /private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib: dlopen(/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib, 0x0001): tried: '/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/lib/librocksdbjni-osx.jnilib' (no such file)
	at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_312]
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950) ~[?:1.8.0_312]
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832) ~[?:1.8.0_312]
	at java.lang.Runtime.load0(Runtime.java:811) ~[?:1.8.0_312]
	at java.lang.System.load(System.java:1088) ~[?:1.8.0_312]
	at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:79) ~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
	at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:57) ~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
	at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:856) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87) ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164) ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
	... 11 more

This issue is tracked by RocksDB community: rocksdb/issues/7720

Issue Links

• duplicates: FLINK-26876 flink-statebackend-rocksdb EmbeddedRocksDBStateBackend crashing on mac m1
• links to: GitHub Pull Request #21747