[SPARK-37272] Add `ExtendedRocksDBTest` and disable RocksDB tests on Apple Silicon

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL, Tests
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 11/Nov/21 00:32
• Updated: 16/Dec/21 22:04
• Resolved: 11/Nov/21 02:39

Description

Java 17 officially support Apple Silicon

• JEP 391: macOS/AArch64 Port
• https://bugs.openjdk.java.net/browse/JDK-8251280

Oracle Java, Azul Zulu, and Eclipse Temurin Java 17 supports Apple Silicon natively.

/Users/dongjoon/.jenv/versions/oracle17/bin/java: Mach-O 64-bit executable arm64
/Users/dongjoon/.jenv/versions/zulu17/bin/java: Mach-O 64-bit executable arm64
/Users/dongjoon/.jenv/versions/temurin17/bin/java: Mach-O 64-bit executable arm64

Since RocksDBJNI still doesn't support Apple Silicon natively, the following failures occur on M1.

$ build/sbt "sql/testOnly *RocksDB* *.StreamingSessionWindowSuite"
...
[info] Run completed in 23 seconds, 281 milliseconds.
[info] Total number of tests run: 32
[info] Suites: completed 2, aborted 2
[info] Tests: succeeded 22, failed 10, canceled 0, ignored 0, pending 0
[info] *** 2 SUITES ABORTED ***
[info] *** 10 TESTS FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.sql.streaming.StreamingSessionWindowSuite
[error] 	org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreIntegrationSuite
[error] Error during tests:
[error] 	org.apache.spark.sql.execution.streaming.state.RocksDBSuite
[error] 	org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreSuite
[error] (sql / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 43 s, completed Nov 10, 2021 4:29:50 PM

This issue aims to add ExtendedRocksDBTest to disable RocksDB selectively on Apple Silicon.

Issue Links

• links to: [Github] Pull Request #34547 (dongjoon-hyun)