[SPARK-24417] Build and Run Spark on JDK11

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 2.3.0
• Fix Version/s: 3.0.0
• Component/s: Build
• Target Version/s: 3.0.0
• Assignee: DB Tsai
• Reporter: DB Tsai
• Created: 29/May/18 23:34
• Updated: 12/Dec/22 18:11
• Resolved: 20/Sep/19 23:16

Description

This is an umbrella JIRA for Apache Spark to support JDK11

As JDK8 is reaching EOL, and JDK9 and 10 are already end of life, per community discussion, we will skip JDK9 and 10 to support JDK 11 directly.

Issue Links

• contains: SPARK-33999 Make sbt unidoc success with JDK11
• is duplicated by: SPARK-24475 Nested JSON count() Exception
• is duplicated by: SPARK-27537 spark-2.4.1/mlib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala.866:Value size is not a member of Object
• is duplicated by: SPARK-27538 sparksql could not start in jdk11, exception org.datanucleus.exceptions.NucleusException: The java type java.lang.Long (jdbc-type='', sql-type="") cant be mapped for this datastore. No mapping is available.
• is duplicated by: SPARK-31531 sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit
• is duplicated by: SPARK-25820 Spark build fails with Java 9
• is related to: SPARK-29173 Benchmark JDK 11 performance with FilterPushdownBenchmark
• is related to: SPARK-26427 Upgrade Apache ORC to 1.5.4
• is related to: SPARK-29194 JDK11 QA
• is required by: SPARK-28596 Use Java 8 time API in date_trunc
• relates to: HADOOP-15338 Java 11 runtime support
• relates to: KAFKA-7264 Initial Kafka support for Java 11
• relates to: PARQUET-1590 [parquet-format] Add Java 11 to Travis
• relates to: HADOOP-10848 Cleanup calling of sun.security.krb5.Config
• relates to: SPARK-28684 Hive module support JDK 11
• relates to: SPARK-25757 Upgrade netty-all from 4.1.17.Final to 4.1.30.Final

Sub-Tasks

• Upgrade to Scala 2.11.12: Resolved
• Upgrade SBT to 0.13.17 with Scala 2.10.7: Resolved
• Upgrade ASM to 6.x to support JDK9+: Resolved
• Upgrade ASM to 7.x to support JDK11: Resolved
• Make Scala 2.12 as default Scala version in Spark 3.0: Resolved
• Accessing sun.misc.Cleaner in JDK11: Resolved
• Add JDK11 in our Jenkins' build servers: Resolved
• install jdk11 on jenkins workers: Closed
• Upgrade to Kafka 2.1.0: Resolved
• Remove deprecated .newInstance(), primitive wrapper class constructor calls: Resolved
• Upgrading Hadoop to 2.7.4 to fix java.version problem: Resolved
• Upgrade Surefire plugin to 3.0.0-M2: Resolved
• Fix core tests for Java 11: Resolved
• Upgrade Mockito to 2.23.4: Resolved
• Work around classloader changes in Java 9 for Hive isolation: Resolved
• Add maven profiles for running tests with JDK 11: Closed
• Upgrade to Kafka 2.1.1: Resolved
• SizeEstimator can't make some JDK fields accessible in Java 9+: Resolved
• Update JPMML to 1.4.8 for Java 9+: Resolved
• Add JAXB reference impl to build for Java 9+: Resolved
• Resolve Scala compiler failure for Java 9+ in REPL: Resolved
• Upgrade Maven to 3.6.1: Resolved
• Upgrade ASM to 7.1: Resolved
• No such method error (sun.nio.ch.DirectBuffer.cleaner()) when reading big table from JDBC (with one slow query): Closed
• No such method error (sun.nio.ch.DirectBuffer.cleaner()): Closed
• Upgrade Surefire plugin to 3.0.0-M3: Resolved
• Change default class loader to ParallelGC: Closed
• Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`: Resolved
• Fix IncompatibleClassChangeError in `FromUnixTime` codegen on JDK9+: Resolved
• Fix Flaky Test: `InputStreamsSuite.Modified files are correctly detected` in JDK9+: Resolved
• on JDK11, IsolatedClientLoader must be able to load java.sql classes: Closed
• Upgrade `xbean-asm7-shaded` to 4.14: Resolved
• Upgrade janino to 3.0.13: Resolved
• Upgrade Mockito to 2.28.2: Resolved
• MultilayerPerceptronClassifierTest.test_raw_and_probability_prediction fails on JDK11: Resolved
• pyspark.mllib.clustering fails on JDK11: Resolved
• Update jersey to 2.27+ (2.29): Resolved
• test_mllib_classification fails on JDK11: Resolved
• Fix checkJavaVersion to accept JDK8+: Resolved
• add java11 support for spark pull request builds: Resolved
• Dependency generation for JDK8/JDK11: Resolved
• Investigate/fix JAXB failure running Pyspark tests on JDK 11: Resolved
• Maven install fails on JDK11: Resolved
• Upgrade scala-maven-plugin to 4.2.0: Resolved
• Add more profiles for JDK8/11 build test: Resolved
• Upgrade Scala to 2.12.10: Resolved
• Update dropwizard metrics to 4.1.1 for JDK 9+ support: Resolved