[SPARK-33772] Build and Run Spark on Java 17

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Build
• Labels: releasenotes
• Target Version/s: 3.3.0
• Assignee: Yang Jie
• Reporter: Dongjoon Hyun
• Created: 14/Dec/20 01:39
• Updated: 05/Jan/23 07:20
• Resolved: 11/Nov/21 17:01

Description

Apache Spark supports Java 8 and Java 11 (LTS). The next Java LTS version is 17.

Version	Release Date
Java 17 (LTS)	September 2021

Apache Spark has a release plan and `Spark 3.2 Code freeze` was July along with the release branch cut.

• https://spark.apache.org/versioning-policy.html

Supporting new Java version is considered as a new feature which we cannot allow to backport.

Issue Links

• is related to: SPARK-35781 Support Spark on Apple Silicon on macOS natively on Java 17
• is related to: SPARK-38396 Improve K8s Integration Tests
• is required by: BEAM-12240 Make Beam compatible with next Java LTS version (Java 17)
• relates to: HADOOP-17177 Java 17 support

Sub-Tasks

• Add Java 17 ea build test to GitHub action: Resolved
• Update commons-lang3 to 3.11: Resolved
• Upgrade checkstyle plugins: Resolved
• Upgrade ORC to 1.7.1: Resolved
• Move Java 17 on GitHub Actions from EA to LTS release: Resolved
• Upgrade ZooKeeper to 3.6.2: Resolved
• Exclude hadoop-client-api transitive dependency from orc-core: Resolved
• Change HiveExternalCatalogVersionsSuite to be able to test with Java 17: Resolved
• Fix Invalid value for HourOfAmPm when testing on JDK 14: Resolved
• Add java.security.jgss/sun.security.krb5 to DEFAULT_MODULE_OPTIONS: Resolved
• Upgrade Scala to 2.12.14: Resolved
• Upgrade Scala to 2.12.15: Resolved
• Upgrade ORC to 1.7.0: Resolved
• Add Daily GitHub Action jobs for Java11/17: Resolved
• Remove explicit GC options from Scala tests: Resolved
• Add `.java-version` to `.gitignore` and `.rat-excludes`: Resolved
• Add Java17 PySpark daily test coverage: Resolved
• Add Java17 SparkR daily test coverage: Resolved
• Document Java 17 support: Resolved
• Upgrade Maven to 3.8.4: Resolved
• Upgrade SBT to 1.6.0: Resolved
• Set file.encoding to UTF-8 for SBT tests: Resolved
• Use static factory methods instead of the deprecated `Byte/Short/Integer/Long` constructors: Resolved
• Disable to_timestamp('366', 'DD') test case: Resolved
• Upgrade Scala 2.13 to 2.13.7: Resolved
• Make dev/mima runs on Java 17: Resolved
• Make sql/core and dependent modules all UTs pass on Java 17: Resolved
• "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17: Resolved
• Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior.: Resolved
• WholeStageCodegenSparkSubmitSuite Failed with Java 17: Resolved
• `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17: Resolved
• Pass all UTs in `mllib-local` and `mllib` with Java 17: Resolved
• Pass all UTs in `repl` with Java 17: Resolved
• Pass all UTs in `external/avro` with Java 17: Resolved
• Pass all UTs in `sql/hive` with Java 17: Resolved
• Pass all UTs in `yarn` with Java 17: Resolved
• Supplement all micro benchmark results use to Java 17: Resolved
• Upgrade RoaringBitmap to 0.9.23: Resolved
• Add InMemoryColumnarBenchmark and StateStoreBasicOperationsBenchmark Java 11/17 result: Resolved
• Try to remove `--add-exports` compile option for Java 17: Resolved
• Add Java 17 support for spark pull request builds: Closed
• Add a vectorized BLAS implementation: Resolved
• Support Java 17 in SparkR SystemRequirements: Resolved
• Support Java 17 in `dev/test-dependencies.sh`: Closed
• Adapt uses of JDK 17 Internal APIs: Closed