[SPARK-43831] Build and Run Spark on Java 21

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Build
• Labels: releasenotes
• Target Version/s: 4.0.0
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 28/May/23 00:22
• Updated: 20/Sep/23 19:47
• Resolved: 20/Sep/23 19:46

Description

• https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html

  JDK version	Minimum Scala versions
  21 (ea)	3.3.1 (soon), 2.13.11, 2.12.18
  20	3.3.0, 2.13.11, 2.12.18
  19	3.2.0, 2.13.9, 2.12.16
  18	3.1.3, 2.13.7, 2.12.15
  17 (LTS)	3.0.0, 2.13.6, 2.12.15
  11 (LTS)	3.0.0, 2.13.0, 2.12.4, 2.11.12
  8 (LTS)	3.0.0, 2.13.0, 2.12.0, 2.11.0

Issue Links

• is related to: SPARK-40732 Make (SQL|ThriftServer)QueryTestSuite pass except Pandas UDF tests in Java 21
• is related to: SPARK-44111 Prepare Apache Spark 4.0.0
• relates to: ORC-1426 Use Java 21-ea instead of 20 in GitHub Action
• links to: Apache Arrow Java 21 Fix
• links to: Scala-Java Compatibility

Sub-Tasks

• Fix constructor for java.nio.DirectByteBuffer for Java 21+: Resolved
• Simplify DirectByteBuffer constructor lookup logic: Resolved
• Remove jdk.incubator.foreign usage in Java 21: Resolved
• Upgrade commons-lang3 to 3.13.0: Resolved
• Upgrade ASM to 9.5: Resolved
• Upgrade Scala to 2.12.18: Resolved
• Upgrade Scala to 2.13.11: Resolved
• Upgrade netty to 4.1.93: Resolved
• Update ORC to 1.9.0: Resolved
• Upgrade Arrow to 13.0.0: Resolved
• Upgrade cyclonedx-maven-plugin to 2.7.9: Resolved
• Add `build_java21.yml` Github Action Daily Job: Resolved
• Add `Utils.isJavaVersionAtLeast21` and make `core` module tests pass with Java 21: Resolved
• Make `catalyst` module passes in Java 21: Resolved
• Make `sql` module passes in Java 21: Resolved
• Make `connect` module pass except arrow-related ones in Java 21: Resolved
• Make HiveExternalCatalogVersionsSuite skip old Spark versions on Java 21: Resolved
• Support Java 21 in SparkR: Resolved
• Fix QueryExecutionErrorsSuite: Resolved
• Enable benchmark on github to support Java 21: Resolved
• Add a Java 21 build task in `build_and_test.yml`: Resolved
• Reenable test_sparkSQL_arrow.R in Java 21: Resolved
• Reenable Arrow-based connect tests in Java 21: Resolved
• Reenable PandasUDF Tests in Java 21: Resolved
• Reenable 'o.a.s.sql.execution.arrow' tests in Java 21: Resolved
• Make `connect-jvm-client` module pass except arrow-related ones in Java 21: Resolved
• Disable PySpark test on the daily test of Java 21 before the new arrow version release: Resolved
• Reenable PySpark test on the daily test of Java 21 after the new arrow version release: Resolved
• Upgrade Jersey to 2.40: Resolved
• Ignore ArrowEncoderSuite for Java 21: Closed
• Support Java 21 in SparkR SystemRequirements: Resolved
• Add Java 21 Maven build to build_and_test.yml: Resolved
• Support G1GC in `spark.eventLog.gcMetrics.*` without warning: Resolved
• Remove System.setSecurityManager usage: Resolved
• Make PySpark compatible with PyArrow 13.0.0: Resolved
• Use Zulu JDK in build_and_test and build_java21 GitHub Action and Java 21: Resolved
• Use Java 21-jre in K8s Dockerfile: Resolved
• Add Java 21 benchmark result: Resolved
• Use ProcessHandle to get pid directly: Resolved
• Upgrade Netty to 4.1.99.Final: Resolved
• Remove outdated benchmark result files `jdk1[17]*results.txt`: Resolved
• Remove deprecated `Runtime.getRuntime.exec(String)` API usage: Resolved
• Remove InaccessibleObjectException usage by using trySetAccessible: Resolved
• Set DEDICATED_JVM_SBT_TESTS in `build_java21.yml`: Resolved
• Upgrade `curator` to 5.6.0: Resolved