[SPARK-45508] Add "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" so Platform can access cleaner on Java 9+

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.2, 3.4.1, 3.5.0
• Fix Version/s: 3.4.2, 4.0.0, 3.5.1, 3.3.4
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 12/Oct/23 00:59
• Updated: 13/Oct/23 17:44
• Resolved: 13/Oct/23 05:30

Description

We need to add `--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED` to our JVM options so that the code in `org.apache.spark.unsafe.Platform` can access the JDK internal cleaner classes.

Issue Links

• links to: GitHub Pull Request #43344