[FLINK-28016] Support Maven 3.3+

• Type: Technical Debt
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.18.0
• Component/s: Build System
• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 13/Jun/22 08:13
• Updated: 23/Jan/24 10:34
• Resolved: 12/May/23 13:43

Description

We are currently de-facto limited to Maven 3.2.5 because our packaging relies on the shade-plugin modifying the dependency tree at runtime when bundling dependencies, which is no longer possible on Maven 3.3+.

Being locked in to such an old Maven version isn't a good state to be in, and the contributor experience suffers as well.

I've been looking into removing this limitation by explicitly marking every dependency that we bundle as optional in the poms, which really means non-transitive. This ensures that the everything being bundled by one module is not visible to other modules. Some tooling to capture developer mistakes were also written.

Overall this is actually quite a nice change, as it makes things more explicit and reduces inconsistencies (e.g., the dependency plugin results are questionable if the shade-plugin didn't run!); and it already highlighted several problems in Flink.

This change will have no effect on users or the released poms, because the dependency-reduced poms will be generated as before and remove all modified dependencies.

Issue Links

• blocks: FLINK-28260 flink-runtime-web fails to execute "npm ci" on Apple Silicon (arm64) without rosetta
• is blocked by: FLINK-28194 Remove workaround around avro sql jar
• is blocked by: FLINK-30089 Remove dependency promotion from kinesis connector
• relates to: FLINK-32734 Add .mvn/maven.config to .gitignore
• relates to: FLINK-34148 Potential regression (Jan. 13): stringWrite with Java8

Sub-Tasks

• table-planner should not use CodeSplitUtils#newName: Closed
• Remove Calcite usages in flink-python: Closed
• Remove netty dependency in flink-test-utils: Closed
• Elasticsearch e2e jars bundle way more than they should: Closed
• Remove netty bundling&relocation in flink-streaming-kinesis-tests: Closed
• Replace IOUtils dependency on oss filesystem: Closed
• Rename hadoop.version property: Closed
• Generalize utils around dependency-plugin: Closed
• Generalize utils around shade-plugin: Closed
• Mark all bundled dependencies as optional: Closed
• Deduplicate Dependency classes across checks: Closed
• Refactor DependencyParser#parseDependencyTreeOutput to return a Tree: Closed
• Store classifier in dependency: Closed
• Notice/ShadeParser parse version/classifier separately: Closed
• Setup conjars https mirror: Closed
• Create new CI image with Maven 3.8.6: Closed
• Include classifier in DependencyTree tree: Closed