[FLINK-23986] Add support for opting-out of Scala

• Type: Technical Debt
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: API / DataSet, API / DataStream

Release Note

The Java DataSet/-Stream APIs are now independent from Scala and no longer transitively depend on it.

The implications are the following:

a)
If you only intend to use the Java APIs, with Java types,
then you can opt-in to a Scala-free Flink by removing the flink-scala jar from the lib/ directory of the distribution.
You are then free to use any Scala version and Scala libraries.
You can either bundle Scala itself in your user-jar; or put into the lib/ directory of the distribution.

b)
If you relied on the Scala APIs, without an explicit dependency on them, then you may experience issues when building your projects.
You can solve this by adding explicit dependencies to the APIs that you are using. This should primarily affect users of the Scala DataStream/CEP APIs.

c)
A large number of modules have lost their Scala suffix.
Any dependency on one of the modules listed below needs to be updated.
Further caution is advised when mixing dependencies from different Flink versions (e.g., an older connector),
as you may now end up pulling in multiple versions of a single module (that would previously be prevented by the name being equal).

Any dependency to one of the following modules needs to be updated to no longer include a suffix:
flink-cep
flink-clients
flink-connector-elasticsearch-base
flink-connector-elasticsearch5
flink-connector-elasticsearch6
flink-connector-elasticsearch7
flink-connector-gcp-pubsub
flink-connector-hbase-1.4
flink-connector-hbase-2.2
flink-connector-hbase-base
flink-connector-jdbc
flink-connector-kafka
flink-connector-kinesis
flink-connector-nifi
flink-connector-pulsar
flink-connector-rabbitmq
flink-connector-testing
flink-connector-twitter
flink-connector-wikiedits
flink-container
flink-dstl-dfs
flink-gelly
flink-hadoop-bulk
flink-kubernetes
flink-runtime-web
flink-sql-connector-elasticsearch6
flink-sql-connector-elasticsearch7
flink-sql-connector-hbase-1.4
flink-sql-connector-hbase-2.2
flink-sql-connector-kafka
flink-sql-connector-kinesis
flink-sql-connector-rabbitmq
flink-state-processor-api
flink-statebackend-rocksdb
flink-streaming-java
flink-table-api-java-bridge
flink-test-utils
flink-yarn

• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 26/Aug/21 06:58
• Updated: 29/Nov/21 10:48
• Resolved: 26/Nov/21 09:43

Description

The goal of this ticket is to fully isolate the Java APIs (excluding the Table API) from Scala, such that jobs that don't require Scala can be run without Scala being in lib/.
In particular we want to allow Scala users that use the Java APIs to use whichever Scala version they want to use.

Issue Links

• is related to: FLINK-13414 Add support for Scala 2.13
• is related to: FLINK-24427 Hide Scala in flink-table-planner from API

Sub-Tasks

• Remove scala references in OneInputStreamTaskTest: Closed
• FieldAccessor has direct scala dependency: Closed
• Remove unused code in TestBaseUtils: Closed
• Pin registration IDs of Java serializers: Closed
• Remove Scala dependencies from flink-cep/flink-streaming-java: Closed
• Separately package Scala-reliant modules: Closed
• Add E2E test: Closed