[SPARK-36731] BrotliCodec doesn't support Apple Silicon on MacOS

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: Build, SQL
• Assignee: Unassigned Assign to me
• Reporter: Dongjoon Hyun
• Created: 13/Sep/21 02:17
• Updated: 16/Jun/22 18:55

Description

[info] Caused by: java.lang.UnsatisfiedLinkError: Couldn't load native library 'brotli'. [LoaderResult: os.name="Mac OS X", os.arch="aarch64", os.version="11.5.2", java.vm.name="OpenJDK 64-Bit Server VM", java.vm.version="25.302-b08", java.vm.vendor="Azul Systems, Inc.", alreadyLoaded="null", loadedFromSystemLibraryPath="false", nativeLibName="libbrotli.dylib", temporaryLibFile="/Users/dongjoon/APACHE/spark-merge/target/tmp/brotli8243220902047076449/libbrotli.dylib", libNameWithinClasspath="/lib/darwin-aarch64/libbrotli.dylib", usedThisClassloader="false", usedSystemClassloader="false", java.library.path="/Users/dongjoon/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."]
[info] 	at org.meteogroup.jbrotli.libloader.BrotliLibraryLoader.loadBrotli(BrotliLibraryLoader.java:35)
[info] 	at org.apache.hadoop.io.compress.BrotliCodec.<init>(BrotliCodec.java:40)

Issue Links

• is related to: SPARK-35781 Support Spark on Apple Silicon on macOS natively on Java 17
• relates to: SPARK-36670 Add end-to-end codec test cases for main datasources