[BEAM-2530] Make Beam compatible with next Java LTS version (Java 11)

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Affects Version/s: Not applicable
• Fix Version/s: 2.29.0
• Component/s: build-system
• Labels: Java11 java9
• Assignee: Ismaël Mejía
• Reporter: Ismaël Mejía
• Created: 28/Jun/17 15:01
• Updated: 19/Jan/22 21:07
• Resolved: 30/Mar/21 19:10

Description

The goal of this task is to validate that the Java SDK and the Java Direct Runner (and its tests) work as intended on the next Java LTS version (Java 11 /18.9). For this we will base the compilation on the java.base profile and include other core Java modules when needed.

Notes:

• Ideally validation of the IOs/extensions will be included but if serious issues are found they will be tracked independently.
• The goal of using the Java Platform module system is out of the scope of this work.
• Support for other runners will be a tracked as a separate effort because other runners depend strongly in the support of the native runner ecosystems.

Issue Links

• Dependent: BEAM-4185 Support building with Java 10
• is blocked by: BEAM-5495 PipelineResources algorithm is not working in most environments
• is blocked by: BEAM-10222 Dataflow sets CPU limits on docker containers near zero so Java 11 only detects 1 CPU
• is blocked by: BEAM-5526 Make ByteBuddyDoFnInvokerFactory injection strategy configurable + drop the singleton
• is related to: BEAM-13506 hcatalog is broken in java 11
• is related to: BEAM-5919 Build breaks when including Multi-release Jars
• is related to: BEAM-5891 Update byte-buddy to 1.9.3
• relates to: BEAM-5061 Invisible parameter type exception in JDK 10
• relates to: BEAM-9283 Add Spark 3 test jobs to the CI (Java 11)
• relates to: BEAM-9796 Add Postcommit Portable ValidatesRunner CI Job for Flink on Java 11
• relates to: MENFORCER-275 Enforcer plugin breaks with JDK 9
• relates to: BEAM-9285 Add Postcommit ValidatesRunner CI Job for Flink on Java 11
• relates to: BEAM-10090 Java 11 Precommit failing tasks
• relates to: BEAM-6704 Fix tests failing while running with Java 11
• relates to: BEAM-9342 Update bytebuddy to version 1.10.8
• relates to: BEAM-3845 Avoid calling Class#newInstance
• requires: BEAM-6630 Upgrade Gradle to 5.2
• links to: Findbugs maven plugin does not support Java 9 yet
• links to: GitHub Pull Request #3460
• links to: GitHub Pull Request #3664
• links to: GitHub Pull Request #3665
• links to: GitHub Pull Request #3903
• links to: GitHub Pull Request #4035
• links to: GitHub Pull Request #4497
• links to: GitHub Pull Request #4662
• links to: GitHub Pull Request #4685
• links to: GitHub Pull Request #8031
• links to: GitHub Pull Request #9994
• links to: GitHub Pull Request #11619
• links to: GitHub Pull Request #11659
• links to: GitHub Pull Request #11691
• links to: GitHub Pull Request #11692
• links to: https://github.com/apache/beam/pull/11619

Sub-Tasks

• The jar files uploaded to maven do not work with Java 9: Resolved
• Make gradle build scripts Java 11 compatible: Resolved
• Invisible parameter type exception in JDK 10: Resolved
• illegal signature attribute when compiling with JDK 11: Resolved
• Support Java 11 in Jenkins: Resolved
• Add a Jenkins job running Java examples on Java 11 Dataflow: Resolved
• Add Automatic-Module-Name headers for Beam Java modules: Resolved
• Project importing Beam that uses Java 11 and JPMS cannot compile: Resolved
• Create ClassLoadingStrategy with Java 11 compatible way: Resolved
• Publish Java 11 SDK Harness docker image: Resolved
• Errorprone 0.0.13 fails during JDK11 build: Resolved
• ParDoSchemaTest fails on Direct Runner running with Java 11: Resolved
• Java 11 ByteBuddyUtils ConvertValueForSetter.convertArray IllegalStateException: Resolved
• Spotbugs failure in JDK11: :sdks:java:core:spotbugsMain (due to spotbugs limitation): Resolved