[FLINK-29866] Split flink-dist

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.20.0
• Component/s: Build System
• Labels: pull-request-available stale-assigned
• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 03/Nov/22 11:46
• Updated: 14/Mar/24 07:36

Description

Users currently have to setup dependencies to various Flink dependencies (APIs like streaming-java, formats, connectors etc), and have to set the scope for each of these correctly w.r.t. what is actually provided by the distribution.
This is unnecessarily tedious and error-prone, made worse by some special-cases like connector-files or the json/csv formats being bundled.

I propose to create a new module flink-dist-pub (name pending!) that acts as a sort-of BOM for what is in the distribution (specifically the flink-dist jar) that is relevant for common DataStream application development.

As follow-ups we could consider creating a table API extension of this module, and publish fat jars for the use-case of thw in FLINK-29497.

Issue Links

• is duplicated by: FLINK-30372 dependencyManagement bom for flink-dist
• is related to: FLINK-30578 Publish SBOM artifacts
• relates to: FLINK-29497 Provide an option to publish the flink-dist jar file artifact
• links to: GitHub Pull Request #21228